System and method for processing using multi core processors, signals, and AI processors from multiple sources to create a spatial heat map of selected region

ABSTRACT

In an example, the present technique includes a method for capturing information from a spatial region to monitor human activities and create a spatial map of the spatial region. In an example, the technique allows a user of a cell phone to move from one location to another location and be tracked using rf backscattering, and each location being identified by the user by communicating a label via a cell phone or other mobile device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. Ser. No. 16/272,975, which ishereby incorporated by reference in its entirety.

BACKGROUND

The present invention relates to techniques, including a method, andsystem, for processing audio, motion, ultra wide band (“UWB”) andfrequency modulated continuous wave (“FMCW”) signals using a pluralityof antenna array, and other conditions and events. In particular, theinvention provides an apparatus using multi-core processors andartificial intelligence processes. More particularly, the inventionprovides a method and system for mapping elements onto a spatiallocation or map with a spatial region. Merely by way of examples,various applications can include daily life, and others.

Various conventional techniques exist for monitoring people within ahome or building environment. Such techniques include use of cameras toview a person. Other techniques include a pendant or other sensingdevice that is placed on the person to monitor his/her movement.Examples include Personal Emergency Response Systems (PERS) devices suchas LifeAlert® and Philips® LifeLine—each of which are just panic buttonsfor seniors to press in case of an emergency. Unfortunately, all ofthese techniques have limitations. That is, each of these techniquesfails to provide a reliable and high quality signal to accurately detecta fall or other life activity of the person being monitored. Many peopleoften forget to wear the pendant or a power source for the pendant runsout. Also, elderly people do not want to look like they are old so oftentimes, elderly people do not wear the pendant.

From the above, it is seen that techniques for identifying andmonitoring a person is highly desirable.

SUMMARY

According to the present invention, techniques related to a method, andsystem, for processing audio, UWB, FMCW signals using a plurality ofantenna array, and other signals and events are provided. In particular,the invention provides an apparatus using multi-core processors andartificial intelligence processes. More particularly, the inventionprovides a method and system for mapping elements onto a spatiallocation or map with a spatial region. Merely by way of examples,various applications can include daily life, and others.

In an example, the present technique includes a method for capturinginformation from a spatial region to monitor human activities and createa spatial map of the spatial region, as further described. In anexample, the technique allows a user of a cell phone to move from onelocation to another location and be tracked using rf backscattering, andeach location being identified by the user by communicating a label viaa cell phone or other mobile device.

In an example, hardware applications can include a stand alone housing,a cable television box, a gaming box, a display device and related box,a robot, a hub, or other apparatus. In an example, the present inventionprovides a system and method for monitoring human activity. The systemhas a stand alone housing, which has a processing platform, anartificial intelligence module, and a plurality of sensing devices,including rf sensors, audio sensors, and motion sensors, each of whichcommunicates information to the artificial intelligence module forprocessing.

The above examples and implementations are not necessarily inclusive orexclusive of each other and may be combined in any manner that isnon-conflicting and otherwise possible, whether they be presented inassociation with a same, or a different, embodiment or example orimplementation. The description of one embodiment or implementation isnot intended to be limiting with respect to other embodiments and/orimplementations. Also, any one or more function, step, operation, ortechnique described elsewhere in this specification may, in alternativeimplementations, be combined with any one or more function, step,operation, or technique described in the summary. Thus, the aboveexamples implementations are illustrative, rather than limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a radar/wireless backscattering sensorsystem according to an example of the present invention.

FIG. 2 is a simplified diagram of a sensor array according to an exampleof the present invention.

FIG. 3 is a simplified diagram of a system according to an example ofthe present invention.

FIG. 4 is a detailed diagram of hardware apparatus according to anexample of the present invention.

FIG. 5 is a simplified diagram of a hub in a spatial region according toan example of the present invention.

FIG. 6 is a simplified diagram of a mini mode in a spatial regionaccording to an example of the present invention.

FIG. 7 is a simplified diagram of a mobile mode in a spatial regionaccording to an example of the present invention.

FIG. 8 is a simplified diagram of a hub device according to an example.

FIG. 9 is a simplified diagram of an ultra-wide band module for the hubaccording to an example of the present invention.

FIG. 10 is a simplified diagram of electrical parameters according to anexample for the ultra-wide band module in the present invention.

FIG. 11 is a simplified system diagram of the ultra-wide band moduleaccording to an example of the present invention.

FIG. 12 is an example of antenna array parameters for the ultra-wideband module according to the present invention.

FIG. 13 is an example of antenna array configuration for the ultra-wideband module according to the present invention.

FIG. 14 is a simplified diagram of FMCW modules and antenna arraysaccording to examples of the present invention.

FIG. 15 is a simplified illustration of three antenna arrays accordingto examples of the present invention.

FIG. 16 is a table illustrating device parameters according to examplesof the present invention.

FIG. 17 is a simplified diagram of a system architecture for an FMCWdevice according to an example of the present invention.

FIG. 18 is a simplified diagram of an alternative system architecturefor an FMCW device according to an example of the present invention.

FIG. 18A is a simplified diagram of various elements in a microcontroller module according to an example of the present invention.

FIG. 19 is a simplified diagram of an alternative system architecturefor an FMCW device according to an example of the present invention.

FIG. 20 is a simplified illustration of each antenna in an arrayaccording to examples of the present invention.

FIG. 21 is a simplified diagram of a processing system according to anexample of the present invention.

FIG. 22 is a simplified diagram of an artificial intelligence moduleaccording to an example of the present invention.

FIG. 23 is a simplified diagram of a processing system according to anexample of the present invention.

FIG. 24 is a simplified diagram of a processing architecture accordingto an example of the present invention.

FIG. 25 is a simplified diagram of a Linux software stack according toan example of the present invention.

FIG. 26 is a simplified diagram of a subsystem of the processing systemin an example of the present invention.

FIG. 27 is a simplified diagram of a boot flow for the presentprocessing system according to an example of the present invention.

FIG. 28 is a simplified diagram of an execution flow for the presentprocessing system according to an example of the present invention.

FIG. 29 is a simplified diagram of a partition among the processingsystem and memory in an example of the present invention.

FIG. 30 is a simplified block diagram of a power management system forthe processing system according to an example of the present invention.

FIG. 31 is a power distribution wiring or network diagram for the systemaccording to an example of the present invention.

FIG. 32 is a simplified diagram of a serial bus interface to couple theradio frequency modules to the processing system according to an exampleof the present invention.

FIG. 33 is a simplified diagram of an Ethernet interface for the presentprocessing system according to an example of the present invention.

FIG. 34 is a simplified diagram of an interface between the processingsystem and memory and storage devices according to examples of thepresent invention.

FIG. 35 is a simplified diagram of an interface between the processingsystem including PCIE interfaces and the AI platform and Wi-Fi sensormodule according to an example of the present invention.

FIG. 36 is a simplified diagram of a WiFi and Bluetooth module coupledto the processing unit in an example of the present invention.

FIG. 37 is a simplified top-view diagram of an audio module according toan example of the present invention.

FIGS. 38 and 39 are respectively a simplified circuit diagram andmicrophone array arrangement according to an example of the presentinvention.

FIG. 40 is a simplified top-view diagram of an inertial sensing moduleaccording to an example of the present invention.

FIG. 41 is a simplified diagram of a user interface according to anexample of the present invention.

FIG. 42 is a simplified block diagram of a cellular module coupled tothe processing system according to an example of the present invention.

FIG. 43 is a simplified diagram of a spatial map configured usingtechniques according to an example of the present invention.

FIG. 44 is a simplified flow diagram of processing information to createa spatial map according to an example of the present invention.

FIG. 45 is a simplified block diagram of a mapping module configuredwith techniques according to an example of the present invention.

FIG. 46 is a simplified illustration of a heat map according to anexample of the present invention.

FIGS. 47, 48, 49, and 50 are illustrations of heat maps according to anexample of the present invention.

FIG. 51 is an illustration of a backscattered signal along withmultipath interference signals in an example.

DETAILED DESCRIPTION OF THE EXAMPLES

According to the present invention, techniques related to a method, andsystem, for processing UWB and FMCW signals using a plurality of antennaarray are provided. In an example, the plurality of antenna array,including a receiving antenna array and a transmitting antenna arrayconfigured to capture and transmit signals in an omni-directionalmanner. In particular, the invention provides an apparatus usingmulti-core processors and artificial intelligence processes. Moreparticularly, the invention provides a method and system for mappingelements onto a spatial location or map with a spatial region. Merely byway of examples, various applications can include daily life, andothers.

FIG. 1 is a simplified diagram of a radar/wireless backscattering sensorsystem 100 according to an example of the present invention. Thisdiagram is merely an example, which should not unduly limit the scope ofthe claims herein. In an example, the system is a wirelessbackscattering detection system. The system has a control line 101coupled to a processing device. The control line is configured with aswitch to trigger an initiation of a wireless signal. In an example, thesystem has a waveform pattern generator 103 coupled to the control line.The system has an rf transmitter 105 coupled to the waveform patterngenerator. The system has transmitting and receiving antenna 107. In anexample, the system has a transmitting antenna coupled to the rftransmitter and an rf receiver 105, which is coupled to an rf receivingantenna. In an example, the system has an analog front end comprising afilter 109. An analog to digital converter 111 coupled to the analogfront end. The system has a signal-processing device 113 coupled to theanalog to digital converter. In a preferred example, the system has anartificial intelligence module 113 coupled to the signal-processingdevice. The module is configured to process information associated witha backscattered signal captured from the rf receiving antenna. Furtherdetails of the present system can be found through out the specificationand more particularly below.

Antenna

In an example, multiple aspects of antenna design can improve theperformance of the activities of daily life (“ADL”) system. For examplein scanning mode the present technique continuously looks for movinghuman targets (or user) to extract ADL or fall. Since these can happenanywhere in the spatial region of a home, the present system hasantennas that have wide field of view. Once the human target isidentified, the technique focuses signals coming only from thatparticular target and attenuate returns from all other targets. This canbe done by first estimating location of the target from our techniqueusing wide field of view antennas and then focusing RF energy on thespecific target of interest once it has been identified. In an example,the technique can either electronically switch a different antenna thathas narrow field of view or could use beam forming techniques tosimultaneously transmit waves from multiple transmit antenna and controltheir phase such that the RF energy constructively builds around thetarget of interest where as it destructively cancels everywhere else.This return will be much cleaner and can boost the performance of ourADL+fall+vital sign sensors.

In another example considers the layout of the antennas itself. In anexample, the technique places transmit and receive antennas in variousdifferent physical configurations (ULA, circular, square, etc.), thatcan help us establish the direction from which the radar signal returns,by comparing phases of the same radar signal at different receivingantennas. The configurations can play a role because differentconfigurations enable direction of arrival measurement from differentdimensions. For example, when the human target falls the vertical angleof arrival changes from top to bottom, therefore a vertical ULA isbetter suited to capture that information. Likewise during walkinghorizontal angle of arrival of the signal varies more therefore it makessense to use horizontal ULA is more sensitive and therefor can provideadditional information for our algorithm. Of course, there can be othervariations, modifications, and alternatives.

RF Unit

In an example, the wireless RF unit can be either pulsed doppler radaror frequency modulated continuous wave (FMCW) or continuous wave doppler(CW). In an example, on the transmit side it will have standard RF unitslike VCO, PLL, among others. On the receive side it can have matchedfilter, LNA, mixer, and other elements. The multiple antennas can beeither driven by a single transmit/receive chain by sharing it in timeor have one each chain for each of the antennas.

Waveform Unit

In an example, waveform pattern generator generates control signals thatdefine the type of radar signal that is generated by the radar RF unit.For example for FMCW, it can generate triangular wave of specific slopeand period, which will linearly sweep the frequency of the RF unitaccording to this parameter. For a pulsed doppler radar, the techniquewill hold generate pulse of specific width and period, which willmodulate the RF output accordingly.

Baseband Unit

In an example, the gain and filter stage filters the radar returns toremove any unwanted signals and then amplifies the remaining signal withdifferent techniques. For example, the present artificial intelligenceor AI technique can determine what target is desirably tracked andprovide feedback to the AI technique, that will filter out radar returnfrom any and all other signals except for the signal that is desirablytracked. If human target is moving the return signal will befluctuating, in that case, the technique applies automatic gain control(AGC) to find the optimal gain, so that entire dynamic range of ADC inthe subsequent stage is satisfied. In an example, the return signal isconverted to digital samples by analog-to-digital converters (ADC),among other front-end elements.

FIG. 2 is a simplified diagram of a sensor array 200 according to anexample of the present invention. This diagram is merely an example,which should not unduly limit the scope of the claims herein. Shown is asensor array. The sensor array includes a plurality of passive sensors201. In an example, the plurality of passive sensors are spatiallydisposed in spatial region of a living area. The sensor array has activesensors, such as one or more radar sensors 203. Additionally, the arrayhas a feedback interface 205, such as a speaker for calling out to ahuman target in the spatial region of the living area.

In an example, the present technique is provided to identify variousactivities in home using non-wearable. In an example, the technique isat least privacy intrusive as possible, and will use sensors that areless intrusive. Examples of sensors can include, without limitation, awireless backscatter (e.g., radar, WiFi.), audio (e.g., microphonearray, speaker array), video (e.g., PTZ mounted, stereo), pressure mats,infrared, temperature, ultraviolet, humidity, pressure, smoke, anycombination thereof, and others.

Active Sensor for RADAR

In an example, the technique can use wireless backscattering to measuremotion of human, a location, and an environmental state, such as dooropening/closing, or other environmental condition. In an example, thewireless backscattering can also be used to measure a vital sign, suchas a heart rate and respiration rate, among others. In an example, thewireless techniques can work in non-line of sight, and is non intrusivecompared to camera or microphone, or others. In an example, thetechnique can use radar\backscatter sensor for two purposes (1) to findthe location of an action; and (2) sense different activities associatedwith the action. Of course, there can be other variations,modifications, and alternatives.

In an example, the present technique and system includes a radar systemthat operates on multiple frequency bands, such as below 10 GHz, around24 GHz, 60 GHz, 77-81 GHz, among others. In an example, differentfrequency interacts differently with various objects in our environment.In an example, available signal bandwidth and permissible signal powerare also regulated differently at different frequency bands. In anexample, the present techniques optimally combine reflections comingfrom a reflector from multiple frequency bands to achieve largecoverage, and/or improve accuracy. Of course, there can be othervariations, modifications, and alternatives.

In an example, each radar is working at a particular frequency band willbe using multiple transmit and receive antennas, as shown. In anexample, using these multiple transmitters, the technique can performtransmit beam forming to concentrate radar signal on a particulartarget. In an example, the technique uses multiple receivers to collectreflected signals coming from various reflectors (e.g., human body,walls). After further processing this will allow us to find thedirection of the reflector with respect to the radar. In an example, thetechnique also uses multiple transmitter and receiver to form virtualarray, this will allow emulate the radar array with large element byusing small number of transmitter and receiver chains. The main benefitis to improve the angle resolution without using a large array, savingspace and component cost. In an example, different antenna arrayconfigurations to improve coverage (using beam forming) or add 3Dlocalization capability (using 2-D array) are included.

In an example using standard radar signal modulation techniques, such asFMCW/UWB, on MIMO radar, the technique will first separate signalscoming from different range and angle. The technique will then identifystatic reflectors, such as chairs, walls, or other features, from movingones, such as human targets, pets, or the like. For moving objects thatare tracked, the technique will further process signals for each of thereflectors. As an example, the technique will use different techniquesto extract raw motion data (e.g., like spectrogram). In an example, thetechnique will apply various filtering process to extract periodicsignals generated by vital signs, such as heart rate, respiration rate,among others. In an example, both the raw motion data and extractedvital signs will be passed to a downstream process, where they arecombined with data from other sensors, such as radar outputs operatingat different frequency or completely different sensors to extract higherinsights about the environment. Of course, there can be othervariations, modifications, and alternatives.

Audio Sensor

In an example, the present technique uses a sensor array that has amultiple microphone array. In an example, these microphones will be useto ascertain the direction of arrival of any audio signal in theenvironment. In an example, the microphone in conjunction with othersensors, such as radar, will be vital in performing two tasks: 1st itwill augment radar signal to identify various activities (walkingproduces a different sound than sitting), if the target is watching TVit is much easier to ascertain it with audio signal; and 2nd in case ofemergency like fall, the technique can use the radar signal to identifythe location of the fall and then beam form microphone array towardsthat location, so that any audio signal produced by the target can becaptured. Of course, there can be other variations, modifications, andalternatives.

Sensor Fusion and Soft Sensors

In addition to a radar sensor, which is consider as active sensors thepresent sensor system (e.g., box, boxes) will also have additionalpassive sensors that captures the sound, chemical signature,environmental conditions. Each of these of the sensors capturesdifferent context about the home that the human being tracking is livingin or occupying. In an example, the UV sensor can monitor how often thesunlight comes in the room. In an example, light sensors determine alighting condition of the human's home or living area.

In an example, a microphone array can have many functions, such as useto sense sound in the room, to figure out how long the human has spentwatching TV, or how many time they went to bathroom by listening to thesound of toilet flushing or other audio signature. In an example, thepresent technique can use creative solutions where it can use the activesensor to find the location of the person and then tune the microphonearray to enhance the sound coming from that location only, among otherfeatures. In an example, the technique can call the sensors that arederived from the hardware sensors using specific algorithms as softwaresensors or soft sensors. So the same hardware sensors can be used formany different applications by creating different software sensors. Herethe software sensors can combine signals from one or more sensors andthen apply sensor fusion and AI techniques to generate the desiredoutput. Of course, there can be other variations, modifications, andalternatives.

Soft Sensor for Detecting Cooking and Eating Habits

In example, radar sensors can determine information about a human'slocation within a home, like if they are in kitchen area, or other. Inan example, when the human target turns on the microphone oven, itgenerates specific RF signature that can be tracked. In an example, thetechnique can combine this information to infer if the human targetwalked to the kitchen and turned on the microphone. Likewise, when thehuman target prepares food in kitchen he/she can make lot of specificnoise like utensils clattering, chopping, or other audio signature. Soif a human target goes to kitchen spends sometime time in the kitchen,and the present microphone pick these sounds, the technique can inferthat food is cooking or other activity.

Soft Sensor for Detecting Bathroom Habits

In an example, toileting frequency can be a very valuable indication ofones wellness. The present technique can track if a human went to thebathroom using the radar or other sensing techniques. In an example,additionally, the technique can pick sound signature of toilet flushing.In an example, the technique combines these two pieces of information,which can be correlated to toileting frequency. In an example,similarly, bathing is a unique activity that requires 4-5 minutes ofspecific movements. By learning those patterns, the technique can figureout ones bathing routines.

Soft Sensor for Detecting Mobile Habits

In an example, different sensors are triggered by different motion of ahuman target. In an example, radar can detect human fall by looking atmicro doppler patterns generating by different part of the target duringfalls. In an example, the technique can also simultaneously hear a fallfrom microphone arrays and vibration sensors. In an example, thetechnique can also detect how pace of movement changes for an individualover a long duration by monitoring the location information provided byradar or other sensing technique. In an example, likewise, the techniquecan gather unstable transfers by analyzing the gait of the target. In anexample, the technique can find front door loitering by analyzing theradar signal pattern. In an example, the technique can figure outimmobility by analyzing the radar return. In this case, the techniquecan figure out the target's presence by analyzing the target's vitalsigns, such as respiration rate or heart rate or by keeping track of thebread crumb of the target's location trace.

In any and all of the above cases, the technique can also learn aboutthe exact environmental condition that triggered a particular state. Forexample, the technique can figure out whether a human target wasimmobile because the target was watching TV or a video for long durationor the target was simply spending a lot of time in their bed. And thesecan be used to devise incentives to change the target's behavioralpattern for better living.

Soft Sensor for Detecting Vital Signs

In an example, the technique can estimate vital signs of a person bysensing the vibration of the target's body in response to the breathingor heart beat, each of the actions results in tiny phase change in theradar return signals, which can be detected. In an example, thetechnique will use several signal processing techniques to extract them.Of course, there can be other variations, modifications, andalternatives.

In an example, different frequency radio wave interact with environmentdifferently. Also phase change due to vital signs (HR,RR) differs byfrequency, for example phase change for a 77 GHz radar is much higherthan for a 10 GHz radar. Thus 77 GHz is more appropriate for estimatingheart-beat more accurately. But higher frequency typically attenuatesmuch more rapidly with distance. Therefore, lower frequency radar canhave much larger range. By using multi-frequency radar in the presenttechnique can perform these vital trade-offs.

Soft Sensor for Detecting Sleeping Habits

In an example, the present radar sensors can detect motions that aregenerated during sleep, such as tossing and turning. In an example,radar sensors can also sense vital signs like respiration rate and heartrate as described earlier. In an example, now combining the pattern oftoss and turn and different breathing and heart beat pattern, thetechnique can effectively monitor the target's sleep. Additionally, thetechnique can now combine results from passive sensors, such as athermometer, UV, photo diode, among others, to find correlation betweencertain sleep pattern and the environmental conditions. In an example,the technique can also use the sleep monitor soft sensor to learn aboutday/night reversal of sleep, and the associated environmental conditionby looking at different passive sensors. In an example, the techniquescan be valuable in providing feedback to improve the human target'ssleep. For example, the technique can determine or learn that certainenvironmental condition results in better sleep and prescribe that toimprove future sleep.

Soft Sensor for Security Applications

In an example, the technique can repurpose many of the sensors describedbefore for security applications. For a security application, thetechnique determines where one or more person is located, which can bedetected using a presence detection sensor that is build on top of radarsignals. In an example, the technique can eliminate one or many falsepositive triggered by traditional security systems. For example, is awindow is suddenly opened by a wind the technique (and system) will lookat presence of human in the vicinity before triggering the alarm.Likewise, combination of vital signs, movement patterns, among others,can be used a biometric to identify any human target. If an unknownhuman target is detected in the vicinity at certain time of the day, thetechnique can trigger an alarm or alert.

In an example, any one of the above sensing techniques can be combined,separated, or integrated. In an example, n addition to radar and audiosensors, other sensors can be provided in the sensor array. Of course,there can be other variations, modifications, and alternatives.

FIG. 3 is a simplified diagram of a system 300 according to an exampleof the present invention. This diagram is merely an example, whichshould not unduly limit the scope of the claims herein. As shown, thesystem has hardware and method (e.g., algorithm), cloud computing,personalized analytics, customer engagement, and an API to variouspartners, such as police, medical, and others. Further details of thepresent system can be found throughout the present specification andmore particularly below.

FIG. 4 is a detailed diagram 400 of hardware apparatus according to anexample of the present invention. This diagram is merely an example,which should not unduly limit the scope of the claims herein. As shown,the hardware units include at least a hub device 401, node 403, andmobile node 405, each of which will be described in more detail below.

In an example, the hub includes various sensing devices. The sensingdevices, include, among others, a radar, a WiFi, a Bluetooth, a Zigbeesniffer, a microphone and speakers, a smoke detector, a temperaturedetector, a humidity detector, a UV detector, a pressure detector, MEMS(e.g., accelerometer, gyroscope, and compass), a UWB sensors (forfinding locations of all the deployed elements relative to each other),among others. In an example, the hub is a gateway to internet via WiFi,GSM, Ethernet, landline, or other technique. The hub also connects toother units (Mini Node/Mobile Node) via Bluetooth, WiFi, Zigbee, UWB andcoordinates them with each other. In an example, certain dataprocessing, such as noise removal, feature extraction to reduce amountof data uploaded to cloud is included. In an example, the hub alone canbe sufficient to cover a small living space. In an example, the hub isdeployed as a single device somewhere in a desirable location (e.g.,middle of the living space) so that it has good connectivity to allother units. An example of such deployment is provided in the Figurebelow.

FIG. 5 is a simplified diagram 500 of a hub in a spatial regionaccording to an example of the present invention. This diagram is merelyan example, which should not unduly limit the scope of the claimsherein. As shown, the hub is deployed in the middle of the living spacein a house.

In an example, as shown in FIG. 6, the system 600 has sensors, which isa subset of sensors in the hub. The sensors are configured to in variousspatial locations to improve coverage area and improve accuracy fordetection of critical events (e.g., fall, someone calling for help). Thesensors also communicate with the hub via WiFi, Bluetooth, ZigBee orUWB, or other technique. Additionally, the sensors or each mini node isdeployed in a bathrooms, where chances of fall is high, a kitchen, wherewe can learn about eating habits by listening to sounds, RF waves,vibrations, or a perimeter of the living space, that will allow us tolearn approximate map of the space under consideration, among otherlocations. Additionally, each of the mini nodes can save power and costsby adding more complexity on the hub. This can even enable us to operateon battery for extended periods. For example, each of the nodes can haveonly single antenna WiFi and hub could have multiple antennas, for WiFibased sensing. Additionally, each of the nodes use simpler radar (e.g.,single antenna doppler) vs MIMO FMCW in the HUB. Additionally, each nodecan be configured with a single microphone whereas the hub can havearray of microphone. Of course, there can be other variations,modifications, and alternatives. As shown, each node is configured in akitchen, shower, perimeter, or other location.

FIG. 7 is a simplified diagram 700 of a mobile node according to anexample of the present invention. This diagram is merely an example,which should not unduly limit the scope of the claims herein. In anexample, each mobile node is a subset of sensors in the hub. The mobilenode sensors include a camera such as RGB or IR. In an example, each ofthe nodes and hub collaboratively figure out interesting events, andpass that information to the mobile node. The technique then goes to thelocation and probes further. In an example, the camera can be useful tovisually find what is going on in the location. In an example, freewillpatrolling can be use to detect anything unusual or to refine details ofthe map created based on perimeter nodes. In an example, onboard UWB canenable precise localization of the mobile node, which can also enablewireless tomography, where the precise RGB and wireless map of theliving space is determined. As shown, the mobile node, such as a mobilephone or smart phone or other movable device, can physically movethroughout the spatial location. The mobile node can also be a drone orother device. Of course, there can be other variations, modifications,and alternatives. Further details of an example of a hub device can befound throughout the present specification and more particularly below.

FIG. 8 is a simplified diagram of a hub device 800 according to anexample of the present invention. As shown, the hub device has acylindrical housing 801 having a length and a diameter. The housing hasan upper top region and a lower bottom region in parallel arrangement toeach other. In an example, the housing has a maximum length of six totwenty four inches and width of no longer than six inches, althoughthere can be other lengths and widths, e.g., diameters. In an example,the housing has sufficient structural strength to stand upright andprotect an interior region within the housing.

In an example, the housing has a height characterizing the housing froma bottom region to a top region. In an example, a plurality of levels803 are within the housing numbered from 1 to N, wherein N is an integergreater than two, but can be three, four, five, six, seven, and others.

As shown, various elements are included. A speaker device 809 configuredwithin the housing and over the bottom region, as shown. The hub devicealso has a compute module 811 comprising a processing device (e.g.,microprocessor) over the speaker device. The device has an artificialintelligence module configured over the compute module, a ultra-wideband (“UWB”) module 813 comprising an antenna array configured over theartificial intelligence module, and a frequency modulated continuouswave (“FMCW”) module 815 with an antenna array configured over the UWCmodule. In an example, the FMCW module being configured to processelectromagnetic radiation in a frequency range of 24 GHz to 24.25 GHz.In an example, the FMCW module outputs an FMCW signal using atransmitter, and receives back scattered signals using a receiver, suchas a receiver antenna. The device has an audio module configured overthe FMWC module and an inertial measurement unit (“IMU”) moduleconfigured over the FMCW module. In an example, the audio modulecomprises a microphone array for detecting energy in a frequency rangeof sound for communication and for detecting a sound energy. In anexample, the IMU module comprises at least one motion detection sensorconsisting of one of a gyroscope, an accelerometer, a magnetic sensor,or other motion sensor, and combinations thereof.

As shown, the speaker device, the compute module, the artificialintelligence module, the UWB module, the FMCW module, the audio module,and the IMU module are arranged in a stacked configuration andconfigured, respectively, in the plurality of levels numbered from 1 toN. In an example, the speaker device comprises an audio outputconfigured to be included in the housing. As shown, the speaker deviceis spatially configured to output energy within a 360 degree range froma midpoint of the device.

In an example, the compute module comprises a microprocessor based unitcoupled to a bus. In an example, the compute module comprises a signalprocessing core, a micro processor core for an operating system, asynchronizing processing core configured to time stamp, and synchronizeincoming information from each of the FMCW module, IMU module, and UWBmodule.

In an example, the device further comprises a real time processing unitconfigured to control the FMCW switch or the UWB switch or other switchrequiring a real time switching operation of less than ½ milliseconds ofreceiving feedback from a plurality of sensors.

In an example, the device has a graphical processing unit configured toprocess information from the artificial intelligence module. In anexample, the artificial intelligence module comprises an artificialintelligence inference accelerator configured to apply a trained moduleusing a neural net based process. In an example, the neural net basedprocess comprises a plurality of nodes numbered form 1 through N.Further details of the UWB module can be found throughout thespecification and more particularly below.

FIG. 9 is a simplified diagram of an ultra-wide band module 900 for thehub according to an example of the present invention. As shown isultra-wide band rf sensing apparatus or module. In an example, theapparatus has at least three antenna arrays 901, 903, 905 configured tosense a back scatter of electromagnetic energy from spatial location ofa zero degree location in relation to a mid point of the device througha 360 degrees range where each antenna array is configured to sense a120 degree range. As shown, each of the three antenna arrays comprises asupport member, a plurality of transmitting antenna 909 spatiallyconfigured on a first portion of the support member. The support memberalso has a transmitting integrated circuit coupled to each of theplurality of transmitting antenna and configured to transmit an outgoingUWC signal. Each of the antenna array also has a plurality of receivingantenna spatially configured on second portion of the support member.The support member also has a receiving integrated circuit coupled toeach of the plurality of receiving antenna and configured to receive anincoming UWB signal and configured to convert the UWC signal into a baseband.

In an example, the device has a triangular configuration comprising afirst antenna array, a second antenna array, and a third antenna arrayincluded in the at least three antenna arrays. The three arrays providea 360 degree visibility range as measured from a horizontal plane, and a80 degree visibility range as measured from a vertical plane normal tothe horizontal plane. As previously noted, the three arrays are enclosedin a housing that provides mechanical support. In an example, each ofthe sensor arrays is provided on a substrate member to be configured inthe triangular configuration. The substrate member has a face arrangedin a normal manner in a direction to each of the support members.

In an example, the UWB module can operate at a center frequency of 7.29GHz and a bandwidth of −1.5 GHz with multiple antenna arrays to achievethe FCC/ETSI compliance standard. In an example, the module has acombined horizontal field-of-view of 360 degrees about a center point ofthe module. In an example, the module has a range greater than 10meters, but can be shorter and longer. In an example, the module isconfigured to achieve a transmission and a receive rate of frames persecond (FPS) equal to or greater than 330 per Tx-Rx. In an example, themodule has a combined horizontal field of view of 360 degrees achievedusing three (3) antenna arrays, each of which covering 120 degrees. Inan example, each antenna array comprises of 1-TX and 4-RX. Each antennaarray is configured to complete the acquisition of a frame within 1millisecond or less. Accordingly, a total of three (3) millisecondscovers all three (3) sectors, achieving a frame rate of 330 fps persector (per Tx-Rx) in an example. In an example, the module hasprogrammability of various parameters similar to Novelda X4M03 module.In an example, the module is a hybrid architecture that has four by fourradar integrated circuit devices in MIMO configuration that switchesbetween the three antenna arrays. The configuration is capable ofsimultaneous capturing of all four Rx frames in an antenna array.Further details of the present UWB module is provided throughout thepresent specification and more particularly below.

FIG. 10 is a simplified diagram 1000 of electrical parameters accordingto an example for the ultra-wide band module. In an example, variousparameters are as listed in the table. Each of the parameters listed aresuggested and can be adjusted to minimize cost and complexity, whilestill achieving performance. In an example, the module has a datatransfer of 3.2 MBps (e.g., 330 fps×200 frame length×2 bytes×2×4receivers×3 modules. In an example, the module can include a microcontroller unit to communicate with X4 SoC through an SPI interface. Inan example, a central processing unit communicates with a compute modulethrough a serial interface such as a universal serial bus, i.e., USB.The micro controller is configured on a board with sufficient memory tostore raw data. In an example, the memory has a capacity of greater than128 MB such as a 128 MB SDRAM. Further details of the electricalparameters configured within a system diagram are provided below.

FIG. 11 is a simplified system diagram 1100 of the ultra-wide bandmodule according to an example of the present invention. As shown, thesystem has a micro controller 1101, such as an integrated circuit soldunder ATSAM4E16E by Microchip Technology Inc. of 2355 West ChandlerBlvd., Chandler, Ariz., USA 85224-6199. The micro controller has aserial interface, such as the universal serial interface, USB. Thecontroller is coupled to random access memory 1105 for storing raw data,and a clock and other miscellaneous circuits 1103. In an example, theoutput of the controller communicates 1107 with four XETHRU X4 SoCsmanufactured by Novelda AS of Norway.

In an example, the basic components of the X4 SoC are a transmitter, areceiver, and related control circuits. The system is controlled by asystem controller and is configurable through a 4(6)-wire serialperipheral interface (SPI). In an example, the X4 receive path (RX)consists of a low noise amplifier (LNA), a digital-to-analog converter(DAC), 1536 parallel digital integrators as well as an output memorybuffer, accessible through the SPI. The RX is tightly integrated withthe transmitter (TX) and is designed for coherent integration of thereceived energy. The X4 transmit path (TX) consists of a pulse generatorcapable of generating pulses at a rate of up to 60.75 MHz. The outputfrequency and bandwidth are designed to fit worldwide regulatoryrequirements. The radar transceiver is able to operate completelyautonomously and can be programmed to capture data at predefinedintervals and then alert or wake up a host MCU or DSP through dedicatedinterrupt pins. A power management unit controls the on-chip voltageregulators and enables low-power applications to use efficient dutycycling by powering down parts of the circuit when they are not needed.The system can be configured to consume less than 1 mW in idle mode whenall analog front end components are turned off. As shown, each of thefour X4 SoCs is coupled in parallel to a switch.

In an example, the switch 1109 is coupled to each antenna array asshown. In an example, the switch can be one listed underHMC241/HMC7992/ADRF5040 SP4T RF Switches of Analog Devices, Inc. Theswitches are non-reflective RF switches from DC to 12 GHz for 4Gcellular, milcom, and radio applications. Examples of HMC241, HMC7992,and ADF5040 are radio frequency (RF) nonreflective/absorptive singlepull, quad throw (SP4T) switches that can interface with 3.3 V, TTL,LVTTL, CMOS, and LVCMOS logic. The switches operate from DC to 12 GHzfrequency range. The HMC241 is a GaAs MMIC RF switch that operates inthe DC to 4 GHz range. The switch takes a single supply at +5 V. TheHMC7992 has a 100 MHz to 6 GHz frequency range. The ESD rating is forthis switch 2 kV (HBM) class 2. The HMC7992 takes a single voltagesupply from ±3.3 V to +5 V. The ADRF5040 comes in a small 4 mm×4 mmLFCSP package and requires a dual ±3.3 V supply. The switch operates inthe 9 kHz to 12 GHz range. The ADRF5040 has the added benefit of being 4kV (HBM) ESD rating. HMC241, HMC7992, and ADF5040 are ideal for 4Gcellular infrastructure such as base stations and repeaters as well asmilitary communications and industrial test and measurementapplications. Of course, there can be other variations, modifications,and alternatives.

In an example, the UWC module comprises a switch configured between aplurality of UWC transceivers. The switch is configured to select one ofthe three antenna arrays to sense the back scatters while the other twoantenna arrays are turned off. In an example, the switch is an rf switchsuch as the one listed under part number ADRF-5040 manufactured byAnalog Devices, Inc. In an example, the UWC module also has a controllerconfigured to control the switch and the three antenna array. In anexample, the controller cycles through a predetermined process to decidewhich one of the three antenna array to activate while the other twoantenna arrays are turned off.

In an example, the at least three antenna array are configured to senseelectromagnetic energy ranging from 6 to 8 GHz in frequency. As noted,the sensing apparatus is spatially positioned within a center of ageographic location of a room to detect movement of human user.

In an example, the present invention provides a method processing anelectromagnetic signal generated from an ultra wide band rf signal todetect an activity of a human user. Referring to FIG. 11, the methodincludes generating a base band outgoing UWC signal from a transmittingintegrated circuit, which is coupled to a micro controller device. Themethod includes transferring and then receiving the base band outgoingUWC signal at a switch device, which is coupled to the micro controller.The switch is configured to direct the outgoing UWC signal using theswitch device to one of three antenna arrays. In an example, the threeantenna array have been configured in a triangular configuration totransmit the outgoing UWC signal from spatial location of a zero degreelocation in relation to a mid point of the device through a 360 degreesvisibility range where each antenna array is configured to sense a 120degree range in a horizontal plane. Each of the antenna array isconfigured to sense and transmit at least an 80 degree visibility rangeas measured from a vertical plane that is normal to the horizontalplane. In an example, each of the three antenna arrays comprise asupport member, a plurality of transmitting antenna spatially configuredon a first portion of the support member, a transmitting integratedcircuit coupled to each of the plurality of transmitting antenna andconfigured to transmit the outgoing UWC signal. Each of the antennaarray also has a plurality of receiving antenna spatially configured onsecond portion of the support member. The antenna array also has areceiving integrated circuit coupled to each of the plurality ofreceiving antenna and configured to receive an incoming UWB signal andconfigured to convert the UWC signal into a base band. In an example,the method also receives a back scattered electromagnetic signal causedby an activity of a human user redirecting the outgoing UWB signal. Inan example, the received signals are processed, using the artificialintelligence module to form an output. Of course, there can be othervariations, modifications, and alternatives.

FIG. 12 is an example 1200 of antenna array parameters for theultra-wide band module according to the present invention. As shown,each antenna array has one 1-Tx and four 4-Rx. Each Tx/Rx is designed tocover 120 degree azimuth field of view and maximize elevation field ofview as desirable. In an example, serial fed patch antennas can be used.In an example, the antennas are fabrication using material such as aRogers 4350 substrate. In an example, the antennas can be an integratedWiFi filter, if desired, optimized for frequencies between 6.0 and 8.5GHz. In an example, the antenna is designed for FCC/ETSI Compliant forTX Center frequency. Of course, there can be other variations,modifications, and alternatives.

FIG. 13 is an example of antenna array configuration 1300 for theultra-wide band module according to the present invention. As shown, theantenna array is spatially provided on a support member, such as aboard. The antenna array comprises four (4) Rx in an antenna array thatare in a two-dimensional (2D) configuration as shown. The Rx4 is alignedwith Rx1, Rx2 or Rx3, and separated by lambda over two, as shown. Eachof the antennas is separated by lambda over two, as shown. Of course,there can be other variations, modifications, and alternatives.

In an example, the present invention provides a method processing anelectromagnetic signal generated from an ultra wide band rf signal todetect an activity of a human user. In an example, the method includesgenerating a base band outgoing UWC signal. The method also includesreceiving the base band outgoing UWC signal at a switch device anddirecting the outgoing UWC signal using the switch device to one ofthree antenna arrays configured in a triangular configuration totransmit the outgoing UWC signal from spatial location of a zero degreelocation in relation to a mid point of the device through a 360 degreesvisibility range where each antenna array is configured to sense a 120degree range in a horizontal plane. Each of the antenna array isconfigured to sense and transmit at least an 80 degree visibility rangeas measured from a vertical plane that is normal to the horizontalplane.

In an example, each of the three antenna arrays has a support member,e.g., board, printed circuit board. In an example, each array has aplurality of transmitting antenna spatially configured on a firstportion of the support member, a transmitting integrated circuit coupledto each of the plurality of transmitting antenna and configured totransmit the outgoing UWC signal, a plurality of receiving antennaspatially configured on second portion of the support member, and areceiving integrated circuit coupled to each of the plurality ofreceiving antenna and configured to receive an incoming UWB signal andconfigured to convert the UWC signal into a base band signal. In anexample, the method includes receiving a back scattered electromagneticsignal caused by an activity of a human user redirecting the outgoingUWB signal.

The apparatus of claim 11 wherein the UWB module comprises a microcontroller unit coupled to a memory resource, and a clock circuit, themicro controller unit being configured with a universal serial businterface coupled to the compute module; wherein the compute module isconfigured with the artificial intelligence module to processinformation from the back scattered electro magnetic signal from thebase band signal to detect the activity of the human entity.

In an example, the support member comprises a major plane positionednormal to a direction of gravity.

In an example, the antenna array comprises at least three antenna arrayspatially arranged in a triangular configuration comprising a firstantenna array, a second antenna array, and a third antenna arrayincluded in the at least three antenna arrays to provide a 360 degreevisibility range as measured from a horizontal plane, and a 80 degreevisibility range as measured from a vertical plane normal to thehorizontal plane. In an example, the antenna array comprises at leastthree antenna array spatially arranged in a triangular configurationcomprising a first antenna array, a second antenna array, and a thirdantenna array included in the at least three antenna arrays to provide a360 degree visibility range as measured from a horizontal plane, and a80 degree visibility range as measured from a vertical plane normal tothe horizontal plane, and further comprising a controller configured tocontrol a switch coupled with each of the three antenna array, thecontroller cycles through a predetermined process to decide which one ofthe three antenna array to activate while the other two antenna arraysare turned off.

In an example, each antenna array comprises 1-TX and 4-RX.

In an example, the system has a switch device coupled between each ofthe antenna array and four receive lanes each of which is coupled to thereceiving integrated circuit device, one transmit lane coupled to atransmitting integrated circuit device, and a micro controller unitcoupled to a bus coupled to the receiving integrated circuit device andthe transmitting integrated circuit device, the micro controller unitcoupled to a memory resource configured with the micro controller tostore raw data from information derived from four receive lanes, themicro controller unit being coupled to a clock.

In an example, each antenna array comprises 1 TX and four RX. In anexample, the system has a switch device coupled between each of thethree antenna arrays and four receive lanes each of which is coupled tothe receiving integrated circuit device, one transmit lane coupled to atransmitting integrated circuit device, and a micro controller unitcoupled to a bus coupled to the receiving integrated circuit device andthe transmitting integrated circuit device, the micro controller unitcoupled to a memory resource configured with the micro controller tostore raw data from information derived from four receive lanes, themicro controller unit being coupled to a clock.

In an example, the present techniques include a method, apparatus, anddevice for processing signals. As shown 1400 in FIG. 14, the presentFMCW device operates at 24 GHz ISM band with multiple antenna arrays1401, 1403, 1405. In an example, the device has various capabilities,such as a combined horizontal field-of-view of 360 degrees, a range of≥12 meters, a FPS equal to or greater than 1000 per Tx-Rx,programmability of various parameters, among other elements. In anexample, each of the antenna array including TX and RX communicates toFMCW modules, as shown. The three antenna array are arranged in atriangular configuration, each of which has a viewing range of 120Degrees.

Referring now to FIG. 15, the device 1500 has various elements, such asantenna array 1, antenna array 2, and antenna array 3. In an example,the device has a 360 degree horizontal field-of-view to be achievedusing three sets of antenna arrays, each covering 120 degrees (as widevertical field-of-view as possible). In an example, each antenna arrayconsists of 2 TX and 4 RX. In an example, the device has an fps of 1000per TX-RX is achieved by generating 6 chirps for the 6 TX sequentiallywithin 1 milliseconds. Of course, there can be other variations,modifications, and alternatives.

As shown in the Table in FIG. 16, various device parameters aredescribed. In an example, the parameters listed are suggested and can bemodified or replaced to minimize cost and complexity, while achievingdesired performance. In an example, sampled radar data are accessed viaUSB interface by a compute module, which is part of the overall system.In an example, the device has a data transfer rate of 6.14 MBps (e.g.,1000 fps×128 samples/frame×2 bytes×8 antenna×3 modules.) In an example,the device has a microcontroller, such as a one from CypressSemiconductor, including a memory resource to store raw radar data. Inan example, the device has a memory that has a capacity of 2 gigabits orgreater. In an example, multiple configurations are described throughoutthe present specification and more particularly below.

In an example, FIG. 17 illustrates a simplified diagram 1700 of a systemarchitecture for the FMCW device according to an example of the presentinvention. In an example, the present system has three antenna array1701 each of which has 2-TX plus 4-RX (i.e., 8 virtual array). Eachantenna array is coupled to a dual channel TX, quad channel RX, quadchannel AFE RX, and FMCW frequency generator 1703. In an example, thesystem has a radio frequency (RF) module including a dual channel TXunder part number ADF5901 by Analog Devices, Inc. In an example, thesystem has a quad channel RX listed under part number ADF5904 by AnalogDevices. The system also has a quad channel AFE RX listed under partnumber ADAR7251 by Analog Devices. Additionally, the system has a FMCWgenerator listed under ADF4159 by Analog Devices. The system has amicrocontroller 1705 listed under part number Cypress MicrocontrollerCYYSB301X, which is coupled to system memory, such as 2 GB-SDRAM, a SPIinterface control between RF module and microcontroller. The system alsohas the microcontroller connected to TCP via a universal serial bus, USB1707. Of course, there can be other variations, modifications, andalternatives.

In an example, FIG. 18 illustrates a simplified diagram 1800 of a systemarchitecture for the FMCW device according to an example of the presentinvention. In an example, the system has three antenna arrays 1801, eachof which has 2-TX+4-RX (i.e., 8 virtual array). In an example, thesystem has an radio frequency module, RF module 1803. The RF module hasa dual channel TX listed under part number ADF5901 by Analog Devices,Inc. The module has a quad channel RX listed under ADF5904 by AnalogDevices.

In an example, the system has a processing and acquisition module 1807.The module has a quad channel AFE RX listed under ADAR7251 by AnalogDevices, and a FMCW generator listed under ADF4159. The module iscoupled to and communicates with a 12 channel—3:1 demux switches 1805listed under TS3DV621 by Texas Instruments Incorporated. The system hasa microcontroller such as a Cypress Microcontroller listed under partnumber CYYSB301X, which is coupled to a memory resource, such as a 2 GBSDRAM. The system has a SPI Interface control between RF module andmicrocontroller. A USB interface is coupled to TCP 1809. Of course,there can be other variations, modifications, and alternatives. Furtherdetails can be found in a more detailed diagram 1850 of FIG. 18A, asdescribed below.

In an example on a transmit lane 1851 referring to FIG. 18A, themicrocontroller is coupled to a wave form generator to output a digitalsignal (e.g., in a register programming) that is converted in an analogto digital converter to a base band analog signal, which is fed to theswitch. The switch is an analog switch that selects between one of thethree arrays. The base band analog in transmitted to an RF integratedcircuit that configures the base band analog into the FMCW rf signal tobe transmitted via the TX antenna.

In an example on a receive lane 1853, four FMCW signals are receivedfrom four RX antenna. The four signals are received in parallel, and fedto and processed in the Rf integrated circuit to output correspondingfour base band analog signals, each of which is fed to the switch. Theswitch allows signals from one of the three antenna array to betransferred to corresponding analog to digital converters, each of whichare in parallel. Each analog to digital converter is coupled to themicrocontroller. Each analog to digital converter configures incomingbase band signal into digital, which is fed to the microcontroller. Ofcourse, there can be other variations, modifications, and alternatives.

In an example, FIG. 19 illustrates a simplified diagram 1900 of a systemarchitecture for the FMCW device according to an example of the presentinvention. The system has three antenna arrays 1901, each of which has2-TX+4-RX (i.e., 8 virtual array). The system has an RF switch 1903 toswitch between any one of the antenna arrays. In an example the systemhas an rf module and acquisition module 1905. The RF module and theacquisition module has a dual channel TX listed under ADF5901 by AnalogDevices. The module has a quad channel RX listed under ADF5904 by AnalogDevices, a quad Channel AFE RX listed under ADAR7251 by Analog Devices,and a FMCW generator listed under ADF4159 by Analog Devices. The modulehas a microcontroller such as the Cypress Microcontroller listed underCYYSB301X by Cypress Semiconductor, Inc. The microcontroller is coupledto a memory resource such as a 2 GB-SDRAM device. The system also has aninterface such as a SPI Interface control 1907 between RF module andCypress microcontroller. The system also has a serial interface such asthe USB interface to connect to TCP. Of course, there can be othervariations, modifications, and alternatives.

FIG. 20 is a simplified example of an antenna array according to anembodiment of the present invention. As shown, serial fed patch antennascan be included. In an example, each antenna array 2001 has 2 TX and 4RX, or can have variations. In an example, each RX covers 120 degreeshorizontal field-of-view. In an example, the Rx has a desirable widevertical field-of-view. In an example, the antenna array has four (4) RXin an antenna array that are equally spaced by lambda over twohorizontally.

In an example, each antenna array has two (2) TX in an antenna arraythat are spaced by lambda apart horizontally and lambda over twovertically to form a virtual 2D array with the 4 RX 2003. In an example,the present virtual antenna mapping is provided to achieve the goal ofpower balancing the physical channels across the multiple physicalantennas especially when multiple input multiple output is deployed inthe downlink. In an example, virtual antenna mapping gives an illusionthat there are actually lesser antennas at the base station than itactually has. The unbalanced balanced power across two transmits pathsare transformed into balanced power at physical antenna ports by virtualantenna mapping. This is achieved using phase and amplitudecoefficients. Thus both the power amplifiers are optimally used even forsignals transmitted on the first antenna. Of course, there can be othervariations, modifications, and alternatives.

In an example, use of higher power with FMCW can be used to capture moregranular features, such as breathing, heart rate, and other small scalefeatures. In an example, lower power and UWB is desirable for more grossfeatures, which has lower frequency. Lower frequency can also penetratewalls, and other physical features.

In an example, the present invention provides an FMCW sensor apparatus.The apparatus has at least three transceiver modules. Each of thetransceiver modules has an antenna array to be configured to sense aback scatter of electromagnetic energy from spatial location of a zerodegree location in relation to a mid point of the device through a 360degrees range where each antenna array is configured to sense a 120degree range. In an example, each of the antenna array has a supportmember, a plurality of receiving antenna, a receiver integrated circuitcoupled to the receiving antenna and configured to receive an incomingFMCW signal and covert the incoming FMCW signal into a base band signal,and a plurality of transmitting antenna. Each antenna array has atransmitter integrated circuit coupled to the transmitting antenna totransmit an outgoing FMCW signal. The apparatus has a virtual antennaarray configured from the plurality of receiving antenna and theplurality of transmitting antenna to form a larger spatial region usingthe virtual antenna array, than a physical spatial region of theplurality of receiving antenna. In an example, the apparatus has atriangular configuration comprising a first antenna array, a secondantenna array, and a third antenna array included in the at least threeantenna arrays to provide a 360 degree visibility range as measured froma horizontal plane, and a 80 degree visibility range as measured from avertical plane normal to the horizontal plane. The apparatus has amaster control board coupled to each of the support members, andconfigured in a normal directional manner with reference to each of thesupport members. The apparatus has a housing enclosing the at leastthree transceiver modules.

In an example, the FMCW sensor apparatus comprises a switch configuredbetween a plurality of FMCW transceivers, such that the switch isconfigured to select one of the three antenna arrays to sense the backscatters while the other two antenna arrays are turned off. In anexample, the antenna array is configured to process electromagneticradiation in a frequency range of 24 GHz to 24.25 GHz.

In an example, apparatus has a controller configured to control theswitch and the three antenna array. In an example, the controller cyclesthrough a predetermined process to decide which one of the three antennaarray to activate while the other two antenna arrays are turned off. Inan example, the three antenna array are configured to senseelectromagnetic energy in a 24 GHz to 24.25 GHz frequency band. In anexample, the sensing apparatus is spatially positioned within a centerof a geographic location of a room to detect movement of human user. Inan example, each of the sensor arrays is provided on a substrate memberto be configured in the triangular configuration.

In an example, the apparatus has a housing. The housing has a maximumlength of six to twenty four inches and width of no longer than sixinches. In an example, the housing has sufficient structural strength tostand upright and protect an interior region within the housing.

In an example, the apparatus has a height characterizing the housingfrom a bottom region to a top region, a plurality of levels within thehousing numbered from 1 to N, and a speaker device configured within thehousing and over the bottom region. In an example, the apparatus has acompute module comprising a processing device over the speaker device,an artificial intelligence module configured over the compute module, aultra-wide band (“UWB”) module comprising an antenna array configuredover the artificial intelligence module, and an audio module configuredover the FMWC module. The apparatus has an inertial measurement unit(“IMU”) module configured over the FMCW module.

In an example, the speaker device, the compute module, the artificialintelligence module, the UWB module, the FMCW module, the audio module,and the IMU module are arranged in a stacked configuration andconfigured, respectively, in the plurality of levels numbered from 1 toN.

In an example, the speaker device comprises an audio output configuredto be included in the housing, the speaker device being configured tooutput energy within a 360 degree range from a midpoint of the device.

In an example, the compute module comprises a microprocessor based unitcoupled to a bus. In example, the compute module comprises a signalprocessing core, a micro processor core for an operating system, asynchronizing processing core configured to time stamp, and synchronizeincoming information from each of the FMCW module, IMU module, and UWBmodule.

In an example, the apparatus has a real time processing unit configuredto control the FMCW switch or the UWB switch or other switch requiring areal time switching operation of less than ½ milliseconds of receivingfeedback from a plurality of sensors. In an example, the apparatus has agraphical processing unit configured to process information from theartificial intelligence module.

In an example, the artificial intelligence module comprises anartificial intelligence inference accelerator configured to apply atrained module using a neural net based process, the neural net basedprocess comprising a plurality of nodes numbered form 1 through N.

In an example, the FMCW module comprises at least three antenna arraysto be configured to sense a back scatter of electromagnetic energy fromspatial location of a zero degree location in relation to a mid point ofthe device through a 360 degrees range where each antenna array isconfigured to sense a 120 degree range.

In an example, each of the antenna arrays comprises a FMCW transceiverand a switch configured between each of the FMCW transceiver and acontroller, such that the switch is configured to select one of thethree antenna arrays and the FMWC transceiver to sense the back scatterswhile the other two antenna arrays are turned off, and furthercomprising a serial interface.

In an example, the audio module comprises a micro phone array fordetecting energy in a frequency range of sound for communication and fordetecting a sound energy.

In an example, the UMU module comprises a support substrate, anelectrical interface provided on the support structure, an accelerometercoupled to the electrical interface, a gyroscope coupled to theelectrical interface, a compass coupled to the electrical interface, aUV detector configured to detect ultraviolet radiation coupled to theinterface, a pressure sensor coupled to the interface, and anenvironmental gas detector configured and coupled to the interface todetect a chemical entity.

In an example, the present invention provides an apparatus forprocessing activities of a human user. The apparatus has an audio moduleand a compute module coupled to the audio module. The apparatus has atransceiver module coupled to the compute module. In an example, thetransceiver module has an antenna array to be configured to sense a backscatter of electromagnetic energy in a frequency range of 24 GHz to24.25 GHz from spatial location of a zero degree location in relation toa mid point of the device through a 360 degrees range where each antennaarray is configured to sense a 120 degree range.

In an example, the antenna array comprises a support member, a pluralityof receiving antenna, a receiver integrated circuit coupled to thereceiving antenna and configured to receive an incoming frequencymodulated continuous wave (FMCW) signal and covert the incoming FMCWsignal into a base band signal, a plurality of transmitting antenna, atransmitter integrated circuit coupled to the transmitting antenna totransmit an outgoing FMCW signal.

In an example, the apparatus has a virtual antenna array configured fromthe plurality of receiving antenna and the plurality of transmittingantenna to form a larger spatial region using the virtual antenna array,than a physical spatial region of the plurality of receiving antenna. Inan example the apparatus has a master control board coupled to thesupport member, and configured in a normal directional manner withreference to the support member and a housing enclosing the transceivermodules, the compute module, and the audio module.

In an example, the present invention has methods using the apparatus,device, and systems. In an example, the method is for processing signalsfrom human activities. The method includes generating an rf signal usinga transceiver module coupled to a compute module and emitting the rfsignal using one of three antenna array and sensing using one of thethree antenna array configured from spatial location of a zero degreelocation in relation to a mid point of the three antenna array through a360 degrees range where each antenna array is configured to sense a 120degree range to capture a back scatter of electromagnetic energy in afrequency range of 24 GHz to 24.25 GHz associated with a human activity.

In an example, the present invention provides an alternative radiofrequency (RF) sensing apparatus. The apparatus has an ultra wide band(UWB) module comprising at least three ultra wide band (UWB) antennaarrays configured in a triangular arrangement to sense a back scatter ofelectromagnetic energy from a spatial location such that the triangulararrangement allows for sensing from a zero degree location in relationto a mid point of the triangular arrangement through a 360 degreevisibility range as measured from a horizontal plane, and a 80 degreevisibility range as measured from a vertical plane that is normal to thehorizontal plane where each UWB antenna array is configured to sense atleast a 120 degree range.

In an example, the apparatus has a frequency modulated continuous wavemodule comprising at least three frequency modulated continuous wave(FMCW) transceiver modules. Each of the FMCW transceiver modules has aFMCW antenna array. In an example, the three FMCW transceiver modulesare configured in a triangular arrangement to sense a back scatter ofelectromagnetic energy from spatial location such that the triangulararrangement allows for sensing from a zero degree location in relationto a mid point of the triangular arrangement through a 360 degreevisibility range as measured from a horizontal plane, and a 80 degreevisibility range as measured form a vertical plane that is normal to thehorizontal plane where each FMCW antenna array is configured to sense atleast a 120 degree range.

FIG. 21 is a simplified diagram of a processing system according to anexample of the present invention. As shown, the processing system has asystem on a chip processing platform, that is a single integratedcircuit chip, including a dual ARM core micro-processing unit, a dualcore digital signal processor, and a dual core image processing unit,among related firmware, interconnections, power management, and otherfeatures. Each of the processing resource is coupled to a bus ormultiple buses.

In an example, the system has multiple interfaces. A USB 3.0 interfacecommunicates to the FMCW module. The I2S interface communicates to theaudio module. A USB 2.0 interface communicates to the UWB module.Another USB 2.0 interface communicates to a user interface, such as akeyboard and a mouse. Other types of serial interfaces can also beincluded. The system also has an RJ-45 and Ethernet interface, a Wi-Fiand Blue Tooth interface, a cellular interface, such as LTE, amongothers. The system has a global positioning sensor interface. The systemhas a power and clock module for power and clocking functions. Thesystem has an inertial measurement unit connector and module. The systemhas multiple PCIE connector interfaces, one of which is coupled to aWi-Fi sensor device. Other features include dynamic random access memoryinterface, embedded multi-media card connection and module, a solid diskdrive connector, and a serial advanced technology attachment connector,among others.

An example of the processing system can be a single integrated circuitchip manufactured by Texas Instruments Incorporated sold as AM572xSitara Arm applications processors. In a datasheet by for the Sitara Armby Texas Instruments, “AM572x devices bring high processing performancethrough the maximum flexibility of a fully integrated mixed processorsolution. The devices also combine programmable video processing with ahighly integrated peripheral set. Cryptographic acceleration isavailable in every AM572x device. Programmability is provided bydual-core Arm Cortex-A15 RISC CPUs with Neon™ extension, and two TI C66xVLIW floating-point DSP cores. The Arm allows developers to keep controlfunctions separate from other algorithms programmed on the DSPs andcoprocessors, thus reducing the complexity of the system software.Additionally, TI provides a complete set of development tools for theArm and C66x DSP, including C compilers, a DSP assembly optimizer tosimplify programming and scheduling, and a debugging interface forvisibility into source code execution.”

The processing system is coupled to a energy source, including a batteryand a plug connection. The system also has a graphical processing moduleor artificial intelligence module for performing processing functionsfrom data received from the interfaces. An example of the processingunit is one sold under the Movidius™ brand by Intel Corporation.

In an example, Movidius provides low-power vision processing solutions,which include the Myriad 2 family of vision processing units (VPUs) plusa comprehensive Myriad Development Kit (MDK), a reference hardware EVMand optional Machine Vision Application Packages. In an example, TheMyriad 2 MA2x5x family of system-on-a-chip (SoC) devices offerssignificant computation performance and image processing capability witha low-power footprint. The Myriad 2 lineup includes the followingproduct configurations: MA2150: 1 Gbit DDR MA2155: 1 Gbit DDR and secureboot MA2450: 4 Gbit DDR MA2455: 4 Gbit DDR and secure boot.

In an example, the Myriad 2 VPUs offer TeraFLOPS (trillions offloating-point operations per second) of performance within a nominal 1Watt power envelope. The Myriad 2 architecture includes enoughperformance to support multiple cameras with flexible image signalprocessing pipelines for each camera, and software programmable visionprocessing with fixed- and floating-point datatypes supported. A robustoverall dataflow design ensures mitigation of processing bottlenecks.

In an example, Myriad 2 MA2x5x incorporates an innovative approach tocombine image signal processing with vision processing. A set ofimaging/vision hardware accelerators supports a world-class ISP pipelinewithout any roundtrips to memory; at the same time they are repurposedto accelerate developers' vision processing algorithms in conjunctionwith a set of special purpose VLIW vision processor cores. Allprocessing elements are tied together with a multi-ported memory thatenables implementation of demanding applications with high efficiency.Further details can be found in a datasheet for Myriad 2 by IntelCorporation, and a simplified block diagram is shown in FIG. 22.

In an example, the artificial intelligence module can process data orinformation by a variety of artificial intelligence techniques. As anexample, machine learning uses on one or more computer algorithms thatimprove automatically through experience. In an example, the techniquecan use unsupervised learning to find patterns in a stream of input,without requiring a human to label the inputs first. Alternatively, thetechnique can use supervised learning that includes both classificationand numerical regression, which requires a human to label the input datafirst. Classification is used to determine what category somethingbelongs in, after seeing a number of examples of things from severalcategories. Regression is the attempt to produce a function thatdescribes the relationship between inputs and outputs and predicts howthe outputs should change as the inputs change. Both classifiers andregression learners can be viewed as “function approximators” trying tolearn an unknown (possibly implicit) function; for example, a statusclassifier can be viewed as learning a function that maps signals from aliving room, for example, to one or more categories such as normal,danger, or other status.

In an example, the artificial intelligence module can provide formachine to interpret data (from any one of the data sources) in a mannerthat is similar to the way humans use their senses to relate to theworld around them. In an example, the artificial intelligence moduletakes in and responds to their environment via each of the sensingmodules, such as radio frequency module, audio module, inertial motionmodule, and others. In an example, the artificial intelligence modulecan also include computer vision, machine hearing, and machine touch.Other examples of artificial intelligence techniques can include, amongothers, natural language processing, deep machine learning, variationsthereof, and combinations, and the like. Of course, there can be othervariations, modifications, and alternatives. See, for example,Wikipedia.com, artificial intelligence.

FIG. 23 is a simplified diagram of a processing system according to anexample of the present invention. As shown, the system has dual ARMcores, each operable at 1.5 GHz, dual digital system processing cores,each operable at 750 MHz, another dual ARM core, each operable at 213MHz, and a programmable realtime unit core, each operable at 200 MHz.The processing system communicates with the processing unit for machinelearning, artificial intelligence, and processing information throughneural networks.

FIG. 24 is a simplified diagram of a processing architecture accordingto an example of the present invention. As shown, the hardware platformincludes ARM processes, digital signal processor, image processingdevice, and programmable realtime unit core. A hardware abstractionlayer is overlying the hardware platform. A layer comprising a bootloader and kernel is overlying the hardware abstraction layer.Applications are configured overlying the boot loader and kernel.Scripts and files overly the applications, as shown.

FIG. 25 is a simplified diagram of a Linux software stack according toan example of the present invention. As shown, the hardware platformincludes ARM processes, digital signal processor, image processingdevice, and programmable realtime unit core. An interface layer isoverlying the hardware platform. Applications are configured overlyingthe interface layer. Scripts and files overly the applications, asshown.

FIG. 26 is a simplified diagram of a subsystem of the processing systemin an example. As shown, the subsystem is generally a firmwarepartition, which has a processing system, including a microprocessorunit running a Linux operating system, a digital signal processor, andan image processing unit. The system also has an artificial intelligencemodule coupled to the processing system via an interface. The Linuxoperating system is stored in a memory resource of dynamic random accessmemory. The system also has a plurality of sensor devices and relatedmodules. The system has Uboot and kernel in memory and a file systemincluded in memory. The file system has firmware for various modulesincluding UWB, FMCW, audio, IMU, IPU, AI, and applications.

FIG. 27 is a simplified diagram of a boot flow for the presentprocessing system according to an example. In an example, the system hasmultiple processor cores—ARM Cortex A15, which is a 32-bit processorcore licensed by ARM Holdings implementing the ARMv7-A architecture,C66x DSP's from Texas Instruments Incorporated, ARM M4 cores, which arefrom a group of 32-bit RISC ARM processor cores licensed by ArmHoldings, as well as an artificial intelligence (AI) subsystem. As anexample, the Cortex A15 runs on Ubuntu Linux and the remote cores (DSP'sand M4's) runs on real time operating systems, called RTOS. In anexample, the AI core runs on Ubuntu Linux and supports a TensorFlow™ byGoogle LLC frame-work. In the normal operation, boot loader (U-Boot/SPL)boots and loads the Cortex A15 with the Ubuntu Linux. The A15 boots theDSP and the M4 cores and enables the PCIE interface to communicate tothe AI core as well. Of course, there can be other variations,modifications, and alternatives.

FIG. 28 is a simplified diagram of an execution flow for the presentprocessing system according to an example. In an example, the ARM CortexA15 controls the execution flow as shown.

FIG. 29 is a simplified diagram of a partition among the processingsystem and memory in an example.

FIG. 30 is a simplified block diagram of a power management system forthe processing system according to an example. As shown, the powermanagement system can be implemented on a single integrated circuit chipsuch as the TPS659037 Power Management Unit manufactured by TexasInstruments Incorporated. In an example, the TPS659037 device is anintegrated power-management IC (PMIC). The device provides sevenconfigurable step-down converters with up to 6 A of output current formemory, processor core, input-output (I/O), or pre-regulation of LDOs.One of these configurable step-down converters can be combined withanother 3-A regulator to allow up to 9 A of output current. All of thestep-down converters can synchronize to an external clock source between1.7 MHz and 2.7 MHz, or an internal fallback clock at 2.2 MHz. TheTPS659037 device contains seven LDO regulators for external use. TheseLDO regulators can be supplied from either a system supply or apre-regulated supply. The power-up and power-down controller isconfigurable and supports any power-up and power-down sequences (OTPbased). The TPS659037 device includes a 32-kHz RC oscillator to sequenceall resources during power up and power down. In cases where a faststart up is needed, a 16-MHz crystal oscillator is also included toquickly generate a stable 32-kHz for the system. All LDOs and SMPSconverters can be controlled by the SPI or I 2C interface, or by powerrequest signals. In addition, voltage scaling registers allowtransitioning the SMPS to different voltages by SPI, I 2C, or roof andfloor control. One dedicated pin in each package can be configured aspart of the power-up sequence to control external resources.General-purpose input-output (GPIO) functionality is available and twoGPIOs can be configured as part of the power-up sequence to controlexternal resources. Power request signals enable power mode control forpower optimization. The device includes a general-purpose sigma-deltaanalog-to-digital converter (GPADC) with three external input channels.The TPS659037 device is available in a 13-pin×13-pin nFBGA package witha 0.8-mm pitch. Of course, there can be other variations, modifications,and alternatives.

FIG. 31 is a power distribution wiring or network diagram for the systemaccording to an example. As shown, the diagram includes the powermanagement system, processing system, and peripheral devices, amongother elements. Each of the elements is coupled to a voltage source froma five volt direct current jack or other power source.

FIG. 32 is a simplified diagram of a serial bus interface to couple theradio frequency modules to the processing system according to anexample. As shown, the diagram shows the processing system, including auniversal serial bus 3.0 and universal serial bus 2.0, and can beothers. In an example, a dual port universal serial bus port couples tothe UWB module and a keyboard and mouse. In an example, the FMCW moduleis coupled directly to the universal serial bus 3.0 interface to theprocessing system.

In an example, the universal serial bus 3.0 has baud rate up to 5 Gbpsin host mode and device mode. Power for the FMCW module is providedusing the universal serial bus interface. In an example, the universalserial bus 2.0 connection has a baud rate of up to 480 Mbps in host modeand device mode and power to the UWB module is provided through theuniversal serial bus 2.0 interface. Of course, there can be othervariations, modifications, and alternatives. In an example, themultiport hub can be a part such as the TUSB8041 Multi-Port USB HUBlisted under http://www.ti.com/lit/ds/symlink/tusb8041.pdf. In anexample, serial bus interface can also includes an auxiliary universalserial bus interface to external peripheral devices for debuggingpurposes, using a keyboard and mouse or other user device.

FIG. 33 is a simplified diagram of an Ethernet interface for the presentprocessing system according to an example of the present invention. Inan example, the interface is a MAC interface, using RGMII. In anexample, the interface connects between Ethernet physical layer and theprocessing system. In an example, the interface also has an Ethernetport to MDI interface through a Category 5 cable. Additionally, theinterface can include a connection between an RF-45 connector and anexternal router for access to the Internet or other network. In anexample, the Ethernet PHY device can be one listed under KSZ9031 RNX byMicrochip Technology Inc., and described underhttp://www.microchip.com/wwwproducts/en/KSZ9031.

FIG. 34 is a simplified diagram of an interface between the processingsystem and memory and storage devices according to examples of thepresent invention. As shown, the memory or storage interfaces caninclude micro SD connector, dynamic random access memory, flash, EEPROM,and storage using SATA. Of course, there can be other variations,modifications, and alternatives.

FIG. 35 is a simplified diagram of an interface between the processingsystem including PCIE interfaces and the AI platform and Wi-Fi sensormodule. As shown, each of the AI platform and Wi-Fi sensor module iscoupled to the processing system using a PCIE edge connector. Each edgeconnector has both transmit and receive lanes to interface to theprocessing system. In an example, the PCIE-II is configured at 5 Gbpsand the PCIE-I is configured at 2.5 Gbps. Of course, there can be othervariations, modifications, and alternatives.

In an example, the PCIE edge connector can be one made by SemtechCorporation, and listed underhttps://www.mouser.com/datasheet/2/418/NG_DS_8-1773459-7_EXPRESS_MINI_CARD_QRG_0816-1260429.In an example, the PCIE reference clock at 100 MHz or greater can be alow noise dual channel 100 MHz Clock generator (CDCM9102) and have 25MHz XTAL input for the 100 MHz Clock generator. In an example, the PCIEinterface to external graphics processing unit, includes an Intel Myriad2485 VPU Card and external peripherals. Of course, there can be othervariations, modifications, and alternatives.

FIG. 36 is a simplified diagram of a WiFi and Bluetooth module coupledto the processing unit in an example of the present invention. As shown,the diagram has a processing system with interfaces, including SDIO,UART, and GPIO. In an example, the diagram also has a 2.4 GHz antennaand 32.768 kHz oscillator. In an example, the module can include aJorjin WG7831-D0 module-2.4 GHz WLAN+Bluetooth based on the WL1831 SoCfrom Texas Instruments. In an example, the module contains a crystal,power amplifier, Tx filter and Tx/Rx switch as well as the necessarypassive components to fully implement the 802.11b,g,n WiFi & Bluetooth4.1 functions. In an example, the WG7831-D0 WLAN is connected to thehost processor via a 1.8V SDIO interface, and the Bluetooth is connectedvia a UART. Of course, there can be other variations, modifications, andalternatives.

FIG. 37 is a simplified top-view diagram of an audio module according toan example of the present invention. In an example, the apparatus has anaudio module, as represented by circularly shaped substrate member. Theaudio module has a microphone array comprising seven microphones,including six peripheral microphones and one center microphoneconfigured and arranged in circular array, although there can be otherconfigurations, quantities, and spatial layouts of the microphones. Inan example, each of the microphones is electrically connect to a dualfour (4) channel analog to digital converter (ADC) with 103 db of signalto noise ratio, or other suitable designs.

In an example, the analog to digital converter uses a bus to connect toa processing system, including a processing device, a signal processor,and other elements. In an example, the ADC uses an I2S interface. In anexample, the I2S interface has been developed by Philips Semiconductor(known today as NXP Semiconductors). In an example, the interface uses apush pull data signal, width of one data line (SD)+2 clock lines (SCK,WS), and a serial protocol. In an example as defined in Wikipedia.com,the “I²S” (Inter-IC Sound), pronounced eye-squared-ess, is an electricalserial bus interface standard used for connecting digital audio devicestogether. In an example, I2S communicates pulse coded modulation (“PCM”)audio data between integrated circuits in an electronic device. In anexample, the I²S bus separates clock and serial data signals, resultingin simpler receivers than those required for asynchronous communicationssystems that need to recover the clock from the data stream.

In an example, the processing system has a digital signal processing(DSP) core, which receives digital audio and performs a beam-formingoperation, including deploying an adaptive spectral noise reductionprocess and the multiple source selection (MSS) process to enhance theaudio quality. In an example, the processing devices, includingmicro-processing unit and audio signal processing unit are provided in aseparate compute module, or other hardware device.

In an example, the multiple source selection processes inputs audioinformation from the plurality of microphones, each of which is sensingan audio signal from a spatial region, in the array directly to the DSPcore, without transferring such data into the processing device, forfaster detection and selection of at least one of the microphone devicesin the array that has the highest audio signal therefrom. Once themicrophone has been selected, the audio information from the selectedmicrophone is outputted or further processed using the processingsystem. In an example, the multiple source selection processes achievesat least a few milliseconds of time off standard processing times, whichoften run through the processor, where the audio information traversesthrough the processing device. As shown, audio signals are captured fromsurroundings, converted to digital signals via A/D converter,transmitted to the digital processing device for audio processing,without traversing the signals through the ARM micro-processing unitcore, as shown.

In an example, the ADC for the audio module has a dedicated I2S channelthat is also interfaced to drive an audio amplifier coupled to aspeaker. In an example, multiple speakers such as dual speakers areintegrated into the apparatus. In an example, the audio amplifier can beone listed under part number TPA3126D2DAD manufactured by TexasInstruments Incorporated, among others. In an example, the driver can bea 50-W, stereo, low-idle-current Class-D amplifier in a thermallyenhanced package. In an example, the driver has a hybrid modulationscheme, which dynamically reduces idle current at low power levels toextend the battery life of portable audio systems (e.g., Bluetoothspeakers, and others). In an example, the Class-D amplifier integratesfull protection features including short circuit, thermal shutdown,overvoltage, under voltage, and DC speaker protection. Faults arereported back to the processor to prevent devices from being damagedduring overload conditions. Other features can also be included.

In an example, the audio module can also include other sensing devices.As an example, the audio module includes an inertial measurement device,a pressure sensor, a gas sensor, and a plurality of LED devices, each ofwhich is coupled to an LED driver. Each of the devices is coupled toauxiliary control hardware, which communicates to a micro-processingunit core using a bus, such as the I2C bus, but can be others.

FIGS. 38 and 39 are respectively a simplified circuit diagram andmicrophone array arrangement according to an example of the presentinvention. As shown, microphone arrays 1-3 couple to an audio analog todigital converter (ADC), which acts as a master, and is coupled to areference clock. As shown, the ADC can be a PCM1864 circular microphoneboard (CMB) from Texas Instruments Incorporated. The ADC is a low-costeasy-to-use reference design for applications that require clear-spokenaudio, such as voice triggering and speech recognition. The ADC designuses a microphone array to capture a voice signal, and converts it to adigital stream that can be used by DSP systems to extract clear audiofrom noisy environments. Microphone arrays 4-6 are coupled to slave ADCdevice, which is coupled to the master ADC device. In an example,digital audio outputs are included and feed digital audio signals into abus, such as the I2S interface, among others. The I2S interface couplesto a computing system, which includes audio output to an audio driver,and speakers.

FIG. 40 is a simplified top-view diagram of an inertial sensing moduleaccording to an example of the present invention. In an example, theapparatus has an inertial motion and sensing module. In an example, themodule has a multi-axis motion sensor. In an example, the sensor can bea part listed under TDK-ICM20948 that provides a 9-axis motion sensorincluding a three (3) axis accelerometer, a magnetometer, a gyroscopeand a digital motion processor. In an example, the module has aninterface that has a slave I2C communication interface to the processingsystem. The module has a master I2C interface to connect to an auxiliarypressure sensor (e.g., Bosch-BMP 180) to perform similar to a ten (10)axis motion sensor.

In an example, the module has an accelerometer, a gyroscope, amagnetometer to form 9-axis inertial motion unit sensor. In an example,these sensors are important to detect the accurate positioning of theapparatus. In an example, the module also provides for additionalinformation regarding the displacement of the apparatus from one spatiallocation to other spatial location.

In an example, the module has a pressure sensor to provide additionalinformation of pressure changes in the surroundings or ambient area. Inan example, the pressure sensor can be configured with the processing todetect opening and/or closing of a door or other building structure.

In an example the module has a gas sensor. In an example, the gas sensoris configured with the processor to detect the amount of carbon monoxideand other toxic gases that can be present in the surroundings where ourdevice is located. In an example, the gas sensor is one sold under thepart number ICM 10020 from TDK or other manufacturers.

In an example, the module has an LED array. In an example, the LED arraycan be a twelve (12) RGBW LED Ring for the Lighting Purposes. LED Driverused such as the one sold under part number LP5569. As shown, the LEDarray is configured spatially around a peripheral region of thesubstrate member, which is circular in this example.

As shown, each of the sensors communicates using the I2C bus, whichcommunicates to various input/output devices on the processing system,as will be described in more detail below. Also shown is a generalpurpose input and output interface coupled to the processing system.

FIG. 41 is a simplified diagram of a user interface according to anexample of the present invention. In an example, the module also has auser interface. An example of an easy to use interface includes buttonssuch as the general purpose input and output (GPIO) buttons configuredon an outer region of the housing. In an example, 4 GPIO push buttonsare placed for multi purpose applications and configured to the housing,and coupled to the processing device. As shown, the buttons include (1)make outgoing call; (2) receive incoming call or mute the A/C audioCODEC; (3) volume up for the A/C audio CODEC; and (3) volume down forthe A/C audio CODEC. Of course, there can be other configurations forthe GPIO buttons.

FIG. 42 is a simplified block diagram of a cellular module coupled tothe processing system. In an example, the cellular module can be anysuitable design, such as one called the U-BLOX LTE Module sold underpart number LARA-R204/SARA-U260, among others. The module can beconfigured to service providers such as AT&T Wireless, Sprint, Verizon,and others. In an example, the module communicates via a universalasynchronous receiver-transmitter (UART) configured for asynchronousserial communication in which the data format and transmission speedsare configurable. The module is also coupled to a removable phone numberSIM card for configuring the system. Of course, there can be othervariations, modifications, and alternatives.

In other examples, a cable television source for digital cabletelevision is coupled to the processing device. In an example, thecoupling can include a co-axial cable or other like designs. In anexample, signal transmission over digital cable television in the UnitedStates can be both 64-QAM and 256-QAM (quadrature amplitude modulation),which is specified in SCTE 07, and is part of the DVB standard (but notATSC). This method carries 38.47 Mbit/s using 256-QAM on a 6 MHzchannel, which can carry nearly two full ATSC 19.39 Mbit/s transportstreams. Each 6-MHz channel is typically used to carry 7-12 digital SDTVchannels (256-QAM, MPEG2 MP/ML streams of 3-5 Mbit/s). On many boxeswith QAM tuners (most notably the DVR boxes), high definition versionsof local channels, and some cable channels are available. Of course,there can be other variations, modifications, and alternatives.

In an example, the present invention provides a system for capturinginformation from a spatial region to monitor human activities. In anexample, the system has a housing, the housing having a maximum lengthof six to twenty four inches and width of no longer than six inches, butcan be other dimensions. In an example, the housing has sufficientstructural strength to stand upright and protect an interior regionwithin the housing, but can include variations. In an example, thehousing has a height characterizing the housing from a bottom region toa top region and a plurality of levels within the housing numbered from1 to N, each of the levels configured with one or more modules.

In an example, the system has an audio module comprising a substratemember and a plurality of peripheral microphone devices spatiallydisposed along a peripheral region of the substrate member. In anexample, each of the peripheral microphone devices has an analog output.In an example, the module has a center microphone device spatiallydisposed within a center region of the substrate member. In an example,the center microphone device has an analog output. In an example, themodule has an analog to digital converter coupled to each of the analogoutputs. The module has a spatial configuration comprising a circularlyshaped region for the peripheral region to provide a 360 degrees fieldof view for the plurality of peripheral microphone devices. A bus deviceis coupled to each of the analog to digital converters. In an example,the bus device communicates with each of the plurality of peripheralmicrophone devices and the center microphone device. The module iscoupled to a signal processor coupled to the bus device. The module iscoupled to a processor device coupled to the signal processing deviceand is configured to process an audio information comprising an audioevent from the plurality of microphone devices using the signalprocessors without transferring the audio information to the processingdevice to achieve a faster selection process of at least onemilliseconds to select one of the microphone devices that has astrongest audio signal, and then transfers the audio information fromthe selected microphone devices. The system also has a cellular networkmodule comprising an interface, which is coupled to the processingdevice. The system has a user interface configured on an exteriorportion of the housing, and coupled to the processor. The user interfaceallows for a user to initiate and make external calls via the cellularnetwork when desirable or also receive external calls from the network.

In an example, the system has other elements. That is, a speaker deviceis coupled to the processor device; and an audio driver device iscoupled to drive the speaker device. In an example, an LED array iscoupled to the processor device. In an example, a plurality of MEMSdevices are coupled to the processor device. In an example, a gas sensordevice is coupled to the processor device. In an example, a pressuresensor device is coupled to the processor device. In an example, theuser interface can be a general purposes input and output device.

In an example, the system has an inertial measurement module comprisingan LED array, an accelerometer device, a gas sensor device, and apressure sensor device configured to detect a pressure within anenvironment of the housing. In an example, the inertial measurementmodule comprising a gas sensor to detect a presence of carbon dioxideand coupled to the processor device configured to send out an alertbased upon a level of carbon dioxide. In an example, the system has aplurality of LED devices configured spatially around a periphery of thesubstrate member to allow for illumination of electromagnetic radiation.In an example, the inertial measurement module comprising a i2C buscoupled to a plurality of LED devices, a gyroscope device, anaccelerometer device, a compass device, a pressure device, and a gassensor, the i2C bus coupled to the processing device. In an example, theprocessing unit comprises an ARM processing unit coupled to a digitalsignal processor and an image processing unit.

Optionally, the system has a network module comprising an interface,which is coupled to the processing device. In an example, the system hasa speaker device coupled to the processor device, and an audio driverdevice coupled to the speaker device, the processer device beingconfigured with the network module to communicate audio information tooutput acoustic energy from the speaker device. The system has a userinterface configured on an exterior portion of the housing, and coupledto the processor.

In an example, the present invention provides a method of capturinginformation from a spatial region to monitor human activities. In anexample, the method uses an apparatus comprising a housing within aspatial region of a living quarter, which is occupied by a human user orusers. In an example, the housing has sufficient structural strength tostand upright and protect an interior region within the housing, thehousing having a plurality of levels within the housing numbered from 1to N. Each of the levels configured with one or more modules, which caninclude any of the ones described herein and others.

In an example, the housing has an audio module comprising: a substratemember; a plurality of peripheral microphone devices spatially disposedalong a peripheral region of the substrate member, each of theperipheral microphone devices having an analog output; a spatialconfiguration using an edge region for the peripheral region to providea 360 degrees field of view from the plurality of peripheral microphonedevices; a bus device coupled to each of the analog to digitalconverters, the bus device communicating with each of the plurality ofperipheral microphone devices; a signal processor coupled to the busdevice; and a micro processor device coupled to the signal processingdevice.

In an example, the method includes sensing a plurality of audio signalscomprising an audio event from each of the plurality of microphonedevices. Each of the plurality of microphone device can be receiving anaudio signal of a different signal strength based upon a spatiallocation of each of the microphone devices. The method includesconverting each of the audio signals from each of the microphone devicesinto a plurality of digital signals in a first format using an analog todigital converter. In an example, the method includes processing thedigital signals in the first format to a second format, which can becompressed or other form to be transported via an interface. The methodincludes transferring the digital signals in the second format using adedicated interface device from each of the plurality of microphonedevices into a receive interface device coupled to the signal processingdevice without transferring the digital signals in the second format tothe micro processing device. The method processes information associatedwith the digital signals using the signal processing device to selectone of the microphone devices that has a strongest audio signal ascompared to any of the other microphone devices; and transfersinformation associated with the digital signals from the selectedmicrophone device to an outgoing interface device. In a preferredexample, the method includes processing the digital signals from theselected microphone device using an artificial intelligence process toidentify the event.

In an example, the present invention provides a system for capturinginformation from a spatial region to monitor human activities. In anexample, the system has a housing. In an example, the housing hassufficient structural strength to stand upright and protect an interiorregion within the housing. In an example, the housing has a heightcharacterizing the housing from a bottom region to a top region. Thehousing has a plurality of levels within the housing numbered from 1 toN, each of the levels configured with one or more modules. In anexample, the system has a processor device comprising a micro-processingunit, a digital signal processing unit, and an image processing unit. Inan example, the system has an audio module comprising a plurality ofmicrophone devices spatially disposed along a region of a substratemember. In an example, each of the microphone devices has an analogoutput configured to the digital signal processing unit with a dedicatedbus. In an example, the system has a radio frequency module comprising aplurality of transmitting antenna and a plurality of receiving antennaand comprising a serial bus coupled to the micro-processing unit. In anexample, the radio frequency module is configured to receive backscattered electromagnetic radiation signals from a spatial region; andthen transmitting information associated with the back scatteredelectromagnetic radiation signals to the processor device. In anexample, the system has a cellular network module comprising a cellularinterface configured to the micro-processing unit with a cellularconnection. A user interface is configured on an exterior portion of thehousing, and is configured with the processor device to initial a callto an external network using the cellular network module. In an example,the system has an artificial intelligence module coupled to theprocessor device using a personal computer bus and configured with atleast the audio module and radio frequency module to process informationfrom at least one of the audio module or the radio frequency module toprocess the information to classify the information into one or moreclassifications; and providing a feedback based upon the classification.

In an example, the system can have other features. In an example, aspeaker device is coupled to the processor device; and an audio driverdevice is coupled to drive the speaker device from a user at a remotelocation communicating through the cellular network module. In anexample, the system has a power management system comprising a power upand down sequence controller, a power good indication, a thermal monitorand protection, a short circuit protection, and a processing healthmonitor, the power management system coupled to the processing deviceand coupled to audio module and the radio frequency module. In anexample, the system has an Ethernet connection coupled to the processingdevice. In an example, the system has a plurality of memory resourcescoupled to the processing device, the memory resources comprising aflash memory array, a dynamic random access memory array, and an EEPROMmemory array. In an example, the system has a WiFi sensor module coupledto the processing device using a personal computer interface. In anexample, the micro-processing unit comprises an ARM processing unitcoupled to the digital signal processing unit and the image processingunit and are configured on a single integrated circuit chip comprisingsilicon bearing material. Of course, there are other variations,modifications, and alternatives.

In an example, the present invention provides a method of capturinginformation from a spatial region to monitor human activities. In anexample, the method includes using an apparatus comprising a housingwithin a spatial region of a living quarter. In an example, the methodincludes sensing a plurality of audio signals comprising an audio eventfrom each of the plurality of microphone devices. In an example, each ofthe plurality of microphone device receives an audio signal of adifferent signal strength based upon a spatial location of each of themicrophone devices. The method includes converting each of the audiosignals from each of the microphone devices into a plurality of digitalsignals in a first format and processing the digital signals in thefirst format to a second format. The method includes transferring thedigital signals in the second format using a dedicated interface devicefrom each of the plurality of microphone devices into a receiveinterface device coupled to the signal processing device withouttransferring the digital signals in the second format to the microprocessing device. The method includes processing information associatedwith the digital signals using the signal processing device to selectone of the microphone devices that has a strongest audio signal ascompared to any of the other microphone devices and transferringinformation associated with the digital signals from the selectedmicrophone device to an outgoing interface device. The method includesprocessing the information using an artificial intelligence module toclassify the information into one of a plurality of classifications.Additionally, the method includes processing the digital signals fromthe selected microphone device and using the artificial intelligencemodule to identify the event from one of the plurality ofclassifications.

In an example, the technique transfers learned information and activityinformation to third parties. The technique teaches itself to learn highlevel behavior that are indicative of a persons welfare using artificialintelligence techniques. In an example, the present technique will thengenerate summary of such activities and send it out to the human's lovedones, caretaker or even emergency response team depending on the urgencyof the situation. For example for regular days, the technique can simplysend short summary like “your mom had a routine activity today”, or “Shewas much less active today.” In an example, where the human has a caretaker visiting few times a week, the technique can send a notificationto them, “It seems she struggles more on yesterday”, so that the caretaker can pay a visit to make sure everything is fine. Alternatively,the technique can be more acute events like fall, shortness ofbreathing, or others, that needs quick attention. In these scenarios,the technique can notify medical response team to provide immediatehelp. Of course, there can be other variations, modifications, andalternatives.

In an example, the present technique can categorize a human target withthe listed ADLs, among others. Examples of ADLs including among others,bathing, brushing teeth, dressing, using toilet, eating and drinking,and sleeping. Other ADLs include preparing meals, preparing drinks,resting, housekeeping, using a telephone, taking medicine, and others.Ambulatory activities including among others walking, doing exercise(e.g., running, cycling), transitional activities (e.g., sit-to-stand,sit-to-lie, stand-to-sit, lie-to-sit in and out of bed or chair), andstationary activities (e.g., sits in sofa, stand for a while, lie in bedor sofa). Of course, there can be other variations, modifications, andalternatives.

In an alternative example, the present technique can determineactivities of a human target with any one of the activities listed. Thelisted activities, including among others, and combinations of goingout, preparing breakfast, having breakfast, preparing lunch, havinglunch, preparing dinner, having dinner, washing dishes, having snack,sleeping, watching TV, studying, having a shower, toileting, having anap, using the Internet, reading a book, shaving, brushing teeth,telephone, listening to music, doing house cleaning, having aconversation, entertain guest, among others.

In an example, the present technique can also identify a rare event. Inan example, the technique identifies when a senior human falls inside ahome with no one around. In an example, the technique is robust, withoutany false negatives. In an example, the technique uses looking atsequence of events that are before to the potential fall and after apotential fall. In an example, the technique combines the contextualinformation to robustly determine if a fall has occurred. Of course,there can be other variations, modifications, and alternatives.

In an example, the technique also detects and measures vital signs ofeach human target by continuous, non-intrusive method. In an example,the vital signs of interest include a heart rate and a respiratory rate,which can provide valuable information about the human's wellness.Additionally, the heart rate and respiratory rate can also be used toidentify a particular person, if more than two target humans living in ahome. Of course, there can be other variations, modifications, andalternatives.

By understanding the context of how the target human (e.g., elderly) isdoing, the technique can also provide valuable feedback directly to theelderly using a voice interface. For example, the technique can sense amood of the human based on sequence of activities and vital signs of thehuman and then ask, “Hi do you want me to call your son”. Based upon thefeedback from the human, the technique can help connect to a third party(or loved one) if their answer is positive. Of course, there can beother alternatives, variations, and modifications.

FIG. 43 is a simplified diagram of a spatial map configured usingtechniques according to an example of the present invention. In anexample, the present technique includes a method for capturinginformation from a spatial region to monitor human activities and createa spatial map of the spatial region, as shown. The method includesplacing an apparatus within a particular location within the spatialregion, which can be within the interior of a building structure, suchas a house, a commercial building, or other region.

In an example, the apparatus includes a housing. In an example, thehousing has sufficient structural strength and protect an interiorregion within the housing. The housing has a height characterizing thehousing from a bottom region to a top region and plurality of levelswithin the housing numbered from 1 to N. In an example, each of thelevels is configured with one or more modules; a processor devicecomprising a micro-processing unit, a digital signal processing unit,and an image processing unit, the processing device provided in one ofthe N levels in the housing; a radio frequency module comprising aplurality of transmitting antenna and a plurality of receiving antennaand comprising a serial bus coupled to the micro-processing unit, theradio frequency module being configured to receive back scatteredelectromagnetic radiation signals from a spatial region; andtransmitting information associated with the back scatteredelectromagnetic radiation signals to the processor device, the radiofrequency module being provided in the housing; and a wireless networkmodule comprising an interface configured to the micro-processing unitwith a wireless connection, the wireless network module being providedin the housing. The apparatus is initiated to turn on the modules.

In an example, the method includes connecting a cellular mobile devicewith the apparatus using the wireless connection or alternatively, thecellular mobile device is connected using a WiFi connection, a cellularconnection, or other connection, through the Internet. In an example,the cellular mobile device is connected to a mapping module, which maybe included in the apparatus or alternatively on a remote server in adata center or other location.

As shown, the method includes moving the cellular mobile device withinthe spatial region while the apparatus remains in a stationary position.As shown, the method allows a particular user of the cellular mobilephone to move through a doorway, through various rooms, and locationswithin the spatial region. In other examples, the cellular mobile devicecan be replaced with other movable devices, such as a tablet computer, alaptop computer, or a robot comprising the cellular mobile device. Ofcourse, there can be other variations, modifications, and alternatives.

In an example, the method includes tracking a location of the cellularmobile device using the back scattered electromagnetic radiationsignals. In an example, the apparatus emits rf electromagnetic radiationsignals which are broad cast throughout the spatial region, one or moreportions of such signals are back scattered off objects within thespatial regions and received by the apparatus. The apparatus processessuch backscattered signals to create an rf backscattered map of thespatial regions, identifying objects, including fixed, moving, andothers.

In an example, concurrently with determining the location of the mobiledevice, the method includes inputting information about the locationusing the cellular mobile device and transferring the informationthrough the wireless connection to the apparatus. In an example, theinformation relates to an ordinary definition of the location. Theordinary definition of the location or related object at the location,and can include, among others, a television, a computer, a bedroom, abathroom, a kitchen, a door, a living room, a garage, a chair, arefrigerator, a sofa, or other object within the spatial region. In anexample, the definition of the location serves as a label for the vectorlocation of the sub-region or object on the spatial region to define thespatial map.

In an example, the method includes storing the information, such as thelabel, about the location into one of a plurality of memory resourcesand storing a location information, such as vector coordinates of thelocation or object location, about the location into one of theplurality of memory resources. In an example, the method includesmapping the information to the location to create a mapping informationand storing the mapping information including the information with thelocation information to create the spatial map. The mapping informationcorrelates, for example, the label with the vector coordinates in themap to create the spatial map.

In an example, the method continues the moving, tracking, inputting,mapping, and storing for other locations numbered from 1 to N withinformation numbered from 1 to N associated with each of the otherlocations to update the spatial map. In an example, the spatial mapincludes a spatial region with various labels identifying each location,such as a bedroom, living room, dining room, or others, and/orcorresponding objects, such as a sofa, table, toilet, door, or otherobject. In an example, the map can be specific for a particular user ofthe mobile device to track the particular user in an example. In anexample, further comprising storing the spatial map for a particularuser of the mobile cellular device; and repeating the moving, tracking,inputting, mapping, and storing for a plurality of locations numberedfrom 1 to N with information numbered from 1 to N associated with eachof the locations to form spatial map for another user. Further detailsof techniques according to examples can be found throughout the presentspecification, and more particularly below.

FIG. 44 is a simplified flow diagram of processing information to createa spatial map according to an example of the present invention. In anexample, the technique also includes method and related system (seefollowing Figure) for capturing information from a spatial region tomonitor human activities. As shown, the flow diagram includes locationinformation such as vector information (e.g., x-y-z, r-theta), andrelated information about the location. The related information is alabel that identifies the location or object. The vector information andrelated label are configured in a digital spatial map, which isdisplayed or output in an alternative manner.

FIG. 45 is a simplified block diagram of a mapping module configuredwith techniques according to an example of the present invention. In anexample, the system has an apparatus including a housing, the housinghaving sufficient structural strength and protect an interior regionwithin the housing. The apparatus has a height characterizing thehousing from a bottom region to a top region; plurality of levels withinthe housing numbered from 1 to N, each of the levels configured with oneor more modules; a processor device comprising a micro-processing unit,a digital signal processing unit, and an image processing unit, theprocessing device provided in one of the N levels in the housing; aradio frequency module comprising a plurality of transmitting antennaand a plurality of receiving antenna and comprising a serial bus coupledto the micro-processing unit, the radio frequency module beingconfigured to receive back scattered electromagnetic radiation signalsfrom a spatial region; and transmitting information associated with theback scattered electromagnetic radiation signals to the processordevice, the radio frequency module being provided in the housing; and awireless network module comprising an interface configured to themicro-processing unit with a wireless connection, the wireless networkmodule being provided in the housing. In an example, one or more of themodules can be included or removed or combined with other modules.

In an example, the system has a mobile device communicating with theapparatus using the wireless connection and configured with the wirelessconnection to allow the mobile device within the spatial region whilethe apparatus remains in a stationary position and the mobile device isconnected to the apparatus using the wireless connection, the mobiledevice being configured with the apparatus to track a location of thecellular mobile device using the back scattered electromagneticradiation signals and to input information about the location using thecellular mobile device and transferring the information through thewireless connection to the apparatus.

In an example, the system has a plurality of memory resources configuredfor storing the information about the location into one of a pluralityof memory resources and storing a location information about thelocation into one of the plurality of memory resources. In an example,the memory resources can be included in the apparatus or coupled to theapparatus at a remote server in a datacenter or cloud. Of course, therecan be other variations, modifications, and alternatives.

In an example, the system has a display device comprising a spatial mapof the spatial region. In an example, the spatial map comprises aplurality of mapping information including the information associatedwith the location for a particular user. That is, the information caninclude a plurality of vector coordinates each of which defines alocation of a particular sub-region (e.g., kitchen, bedroom, bathroom)or object, such as a chair, table, toilet, door, or others. In anexample, the system has a mapping module coupled to the display deviceand the one or more memory resources, and configured to receiveinformation about the location, and the location information about thelocation; and transferring the information about the location and thelocation information on the spatial map. In an example, the mappingmodule can include a digital map, which is stored in memory, a pluralityof handlers for receiving and outputting vector coordinates, each ofwhich is a location of a region or object. The mapping module can alsolink each vector coordinate with a label identifying the region orobject. Of course, there can be other variations, modifications, andalternatives.

In an example, the invention provides a system for capturing informationfrom a spatial region to monitor human activities. In an example, thesystem provides for a heat map of a spatial region of user activity. Thesystem has an apparatus including a housing, which has sufficientstructural strength and protect an interior region within the housing.In an example, the housing has a length characterizing the housing froma bottom region to a top region; plurality of regions within the housingnumbered from 1 to N, each of the regions configured with one or moremodules.

In an example, the system has a processor device comprising amicro-processing unit, a digital signal processing unit, and an imageprocessing unit. In an example, the processing device provided in one ofthe N regions in the housing; The apparatus has a radio frequency modulecomprising a plurality of transmitting antenna and a plurality ofreceiving antenna and comprising a serial bus coupled to themicro-processing unit. In an example, the radio frequency module isconfigured to receive back scattered electromagnetic radiation signalsfrom a spatial region; and transmitting information associated with theback scattered electromagnetic radiation signals to the processordevice, the radio frequency module being provided in the housing.

In an example, the system has a wireless network module comprising aninterface configured to the micro-processing unit with a wirelessconnection, the wireless network module being provided in the housing.In an example, the system has a plurality of memory resources configuredfor storing the information about the location into one of a pluralityof memory resources and storing a location information about thelocation into one of the plurality of memory resources.

In an example, the system has a mapping module coupled to the displaydevice and the one or more memory resources, and configured to receiveinformation about the location, and the location information about thelocation; and transferring the information about the location and thelocation information on the spatial map and configured to removemultipath interference information associated with the particular userand one or more events associated with the spatial region.

In an example, the system has a display device comprising a spatial heatmap of the spatial region, the spatial heat map comprising a pluralityof mapping information including the information associated with thelocation for a particular user, and filtered to remove a multipathinterference information associated with the particular user and one ormore events associated with the spatial region.

In an example, the system has a cellular module coupled to the processordevice for communicating to the apparatus. In an example, the cellularmobile device is a cellular phone, a tablet computer, a laptop computer,or a robot comprising the cellular mobile device, or other suitabledevice. In an example, the RF vision processor comprises a plurality ofnodes, each of which is tuned with a weighing factor.

In an example, the mapping module is configured to mapping an eventassociated with the one or more objects or one or more living quartersonto a digitized representation of the spatial region. In an example,the mapping module using a filter to remove the multipath interferenceinformation. In an example, the mapping module is configured to removingan event associated with the one or more objects or one or more livingquarters from the spatial heat map. Further details of techniquesaccording to the present example can be found throughout the presentspecification and more particularly below.

In an example referring to FIG. 46, the present invention provides amethod for creating a heat map of a spatial location for identifyingevents within a time period, which can be a minute, hour, a day, a week,or month or a year, depending upon the application. The method includesgenerating an electromagnetic radio frequency (RF) signal from an RFgenerating device and emitting the electromagnetic RF signal into aspatial region. In an example, as shown, the spatial region comprises aliving area, including a plurality of objects and one or more livingquarters. In an example, the living area can include, among others, abedroom, a living room, a bathroom, a closet, a dining room, a garage, ahallway, and other living quarters. In an example, the objects caninclude, among others, a chair, a bed, a desk, a toilet, a window, adoor, a table, a mirror, a television, a computer, or any other itemthat is suitable for a living area, which can generate a false positivesignal based upon the back scattered RF signal being detected.

In an example, the method includes detecting a back scattered RF signal,using an RF receiving antenna, from the spatial region and convertingthe back scattered RF signal from an RF analog signal into an RF digitalsignal. In an example, the method includes transferring informationrelated to the RF digital signal, using a serial bus, to amicro-processing device coupled to an artificial intelligence processordevice comprising a natural language processor and an RF visionprocessor. The method includes processing information associated withthe RF digital signal using RF vision processor included in theartificial intelligence processor and identifying an event numbered from1 to N, where N is an integer greater than 5, associated with one ormore of the objects or one or more living quarters within the timeperiod based upon the processed information from the RF visionprocessor.

In an example, the method includes mapping, using a mapping module, theevent associated with the one or more objects or one or more livingquarters onto a digitized representation of the spatial region. Themethod then performs steps of repeating the generating, emitting,detecting, transferring, processing, identifying, and mapping for otherevents numbered from 2 through N onto the digitized representation ofthe spatial map to create a spatial and time based heat map of thespatial region.

In an example, the event is associated with an interference signal; andfurther comprising removing the plurality of events to reduce multipathinterference. In an example, the method then performs steps of repeatingthe generating, emitting, detecting, processing, and identifying a usersignal from a user, while removing the plurality of events to reducemultipath interference that causes interference with the user signal.Further details of the present technique can be found throughout thepresent specification and more particularly below.

FIGS. 47, 48, and 49 are illustrations of heat maps according to anexample of the present invention. To prove the operation of the presentinvention, experiments were performed to generate the heat maps. Asshown, FIG. 47 illustrates a heat map of a particular user and a livingarea that has not been filtered. FIG. 48 illustrates a heat map of theparticular user and the living area that has been filtered usingnormalization. FIG. 49 is an activity of the particular user over aselected time period, where the activity has not been filtered. FIG. 50is a series of view of a plurality of heat maps for different timeperiods for the particular user according to an example. As shown, theexperiments demonstrates that a heat map can be created and improvedusing a filtering technique.

FIG. 51 is a illustration of a backscattered signal along with multipathinterference signals in an example. As shown, the main signal isassociated with a backscattered rf signal from a moving person, whichmoves back and forth, creating a sinusoidal curve. The other points thatare spatially separate and appear someone random are multipathinterference signals. As shown, the multipath interference signalfollows the main signal but are spatially separated in the illustration.That is, the multipath interference signals have been translated. Avariety of signals are shown. According to this example, the multipathinterference can be filtered and removed thereby creating a much cleanerdetection process without the interference.

In an example, the present technique provides a method an system thatimproves via learning over a time frame, such as a day, weeks, or othertime period. In an example, the technique takes into account an indoorenvironment effect on the outgoing and backscattered radar signals toachieve better localization based on unique environment fingerprints,enhancing detection by inverting the environment effect, among otherbenefits.

In an example, as used herein, the term multipath can be a convolutionprocess of features of the environment with returns from targets. In anexample, first and second order multipath is dominating the indoorspace. In an example, different locations impose very differentmultipath, even at different spatial (e.g., x-y-z) locations at sameroom. In an example, multipath is generated by environment features,that are assumed to be a random process with slow fluctuation over time.In an example, multipath effect creates ghost targets/signals thatinterfere/overlap with the real target/signal. Multipath leads todegrading both localization and behavior detection/classification. Theseand other limitations are identified and overcome using presenttechniques.

Having described various embodiments, examples, and implementations, itshould be apparent to those skilled in the relevant art that theforegoing is illustrative only and not limiting, having been presentedby way of example only. Many other schemes for distributing functionsamong the various functional elements of the illustrated embodiment orexample are possible. The functions of any element may be carried out invarious ways in alternative embodiments or examples.

Also, the functions of several elements may, in alternative embodimentsor examples, be carried out by fewer, or a single, element. Similarly,in some embodiments, any functional element may perform fewer, ordifferent, operations than those described with respect to theillustrated embodiment or example. Also, functional elements shown asdistinct for purposes of illustration may be incorporated within otherfunctional elements in a particular implementation. Also, the sequencingof functions or portions of functions generally may be altered. Certainfunctional elements, files, data structures, and so one may be describedin the illustrated embodiments as located in system memory of aparticular or hub. In other embodiments, however, they may be locatedon, or distributed across, systems or other platforms that areco-located and/or remote from each other. For example, any one or moreof data files or data structures described as co-located on and “local”to a server or other computer may be located in a computer system orsystems remote from the server. In addition, it will be understood bythose skilled in the relevant art that control and data flows betweenand among functional elements and various data structures may vary inmany ways from the control and data flows described above or indocuments incorporated by reference herein. More particularly,intermediary functional elements may direct control or data flows, andthe functions of various elements may be combined, divided, or otherwiserearranged to allow parallel processing or for other reasons. Also,intermediate data structures of files may be used and various describeddata structures of files may be combined or otherwise arranged.

In other examples, combinations or sub-combinations of the abovedisclosed invention can be advantageously made. The block diagrams ofthe architecture and flow charts are grouped for ease of understanding.However it should be understood that combinations of blocks, additionsof new blocks, re-arrangement of blocks, and the like are contemplatedin alternative embodiments of the present invention.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

The invention claimed is:
 1. A method for creating a heat map of aspatial location for identifying events within a time period, the methodcomprising: generating an electromagnetic radio frequency (RF) signalfrom an RF generating device; emitting the electromagnetic RF signalinto a spatial region, the spatial region comprising a living area,including a plurality of objects and one or more living quarters;detecting a back scattered RF signal, using an RF receiving antenna,from the spatial region; converting the back scattered RF signal from anRF analog signal into an RF digital signal; transferring informationrelated to the RF digital signal, using a serial bus, to amicro-processing device coupled to an artificial intelligence processordevice comprising a natural language processor and an RF visionprocessor; processing information associated with the RF digital signalusing the RF vision processor included in the artificial intelligenceprocessor; identifying an event numbered from 1 to N, where N is aninteger greater than 5, associated with one or more of the objects orone or more living quarters within the time period based upon theprocessed information from the RF vision processor; mapping the eventassociated with the one or more objects or one or more living quartersonto a digitized representation of the spatial region; and repeating thegenerating, emitting, detecting, transferring, processing, identifying,and mapping for other events numbered from 2 through N onto thedigitized representation of the spatial region to create a spatial andtime based heat map of the spatial region.
 2. The method of claim 1wherein the event is associated with an interference signal; and furthercomprising removing the plurality of events to reduce multipathinterference.
 3. The method of claim 1 wherein the object is selectedfrom movement from a door, a window, a mirror, or other objects thatcause a false positive signal.
 4. The method of claim 1 furthercomprising repeating the generating, emitting, detecting, processing,and identifying a user signal from a user, while removing the pluralityof events to reduce multipath interference that causes interference withthe user signal.
 5. The method of claim 1 wherein the time period rangesfrom one day to one week.
 6. The method of claim 1 wherein the RF signalcan be selected from a UWB signal, a FMCW signal, or combination of theUWB signal and the FMCW signal.
 7. The method of claim 1 wherein the RFsignal is transmitted by one of a plurality of RF modules configured ina triangular arrangement capable of transmitting the RF signal within a360 degree range from a center region of the triangular arrangement. 8.The method of claim 1 wherein the RF vision processor comprises aplurality of nodes, each of which is tuned with a weighing factor. 9.The method of claim 1 further comprising normalizing the heat map toremove multi-path interference and output a normalized heat map for aparticular user.
 10. The method of claim 9 further comprising removingmultipath interference information associated with a particular user andthe one or more events.
 11. A system for capturing information from aspatial region to monitor human activities, the system comprising: anapparatus including a housing, the housing having sufficient structuralstrength and protecting an interior region within the housing, thehousing having a length characterizing the housing from a bottom regionto a top region; a plurality of regions within the housing numbered from1 to N, each of the regions configured with one or more modules; aprocessor device comprising a micro-processing unit, a digital signalprocessing unit, and an image processing unit, the processing deviceprovided in one of the N regions in the housing; a radio frequencymodule comprising a plurality of transmitting antenna and a plurality ofreceiving antenna and comprising a serial bus coupled to themicro-processing unit, the radio frequency module being configured toreceive back scattered electromagnetic radiation signals from a spatialregion, and transmit information associated with the back scatteredelectromagnetic radiation signals to the processor device, the radiofrequency module being provided in the housing; and a wireless networkmodule comprising an interface configured to the micro-processing unitwith a wireless connection, the wireless network module being providedin the housing; a plurality of memory resources configured for storinginformation about a location into one of the plurality of memoryresources and storing s-location information about the location into oneof the plurality of memory resources; a mapping module coupled to adisplay device and one or more of the plurality of memory resources, andconfigured to receive the information about the location, and thelocation information about the location; and transferring theinformation about the location and the location information on thespatial region and configured to remove multipath interferenceinformation associated with a particular user and one or more eventsassociated with the spatial region; and a display device comprising aspatial heat map of the spatial region, the spatial heat map comprisinga plurality of mapping information including the information associatedwith the location for the particular user, and filtered to remove themultipath interference information associated with the particular userand the one or more events associated with the spatial region.
 12. Thesystem of claim 11 further comprising a cellular module coupled to theprocessor device for communicating to the apparatus.
 13. The system ofclaim 11 further comprising a cellular module coupled to the processordevice for communicating to the apparatus, wherein the cellular moduleis a cellular phone, a tablet computer, a laptop computer, or a robotcomprising a cellular mobile device.
 14. The system of claim 11 whereinthe processor device comprises a plurality of nodes, each of which istuned with a weighing factor.
 15. The system of claim 11 wherein themapping module is configured to mapping an event associated with the oneor more objects or one or more living quarters onto a digitizedrepresentation of the spatial region.
 16. The system of claim 15 whereinthe mapping module uses a filter to remove the multipath interferenceinformation.
 17. The system of claim 11 wherein the mapping module isconfigured to remove an event associated with one or more objects or oneor more living quarters from the spatial heat map.
 18. The system ofclaim 17 wherein the wireless connection is provided through an Internetconnection.